AI News Aggregator and Automatic News Scraping from Multiple Sources

A morning routine you might find frustrating is opening up your web browser and discovering more than 15 news-related tabs open on your screen. You’ll browse through TechCrunch, look at VentureBeat, then go to the MIT Review — all in about an hour. Meanwhile, your competitors will have already consumed all the information and made a decision based on the information they’ve processed. This is where an automated AI news aggregation solution comes into play. The AI news aggregator does the grunt work of collecting, filtering, and structuring the data for you. When you log on in the morning, there will be one file that contains everything you need for your business and your life; it will be organized, readable, and concise.

During my eight years of working in Cryptocurrencies and Artificial Intelligence, I learned a very valuable lesson: the person who gets, processes and analyzes all the information first is the winner and makes the money. Everyone else is just trying to catch up. Automating the reading of the news is less about convenience and more about money!

So what is an AI News Aggregator, and why do you need one?

An AI News Aggregator is simply an electronic robot that crawls the Internet for news articles regarding the field of Artificial Intelligence and collects all of it in one location. It will collect the information based on any criteria you decide — removing the "junk" and leaving only the real story.

The AI News Aggregator will act as your "personal analyst" and will work for you 24 hours a day, 7 days a week without sleeping, and without ever missing important information.

A good aggregator will:

• Be able to automatically collect data from sources every 15 minutes, every hour, or once every day (based on your needs). Web scraping enables the extraction of data from web pages without the availability of an API. The parsing program retrieves contents (i.e. headlines, descriptions, dates, etc.) from the source’s HTML page on its own.

• Connect to Many Sources – An aggregator may connect to as many as 20 different locations: news sites (such as TechCrunch), bloggers about artificial intelligence or technology, scientific archives (like arXiv), social media (Facebook / Twitter/X and Reddit) and the official blogs from OpenAI and Google AI, all together providing an extensive view of the marketplace.

• Content Filtering and Prioritization – Duplicates/Removals of Unwanted Data is done by retrieving; filtering of data by keyword (machine learning, GPT, computer vision), source or virality. Therefore out of the hundreds of news sources, you’d only find 10-15 articles really worth your time.

• Markdown Export – You’ll receive documentation in a well-organized format. Markdown files are easy to read and use with platforms such as Obsidian or Notion allowing your the creation of orderly referencing of things you learn while researching.

Cost/Time Reductions

Clients that use automated aggregation for content curation for analysis can save 60-70% from market research time and accelerate strategic decision making by 40%. Also, if you are active as a trader/investor, a single hour of time can result in the market breaking a trend or missing an opportunity. Do you spend approximately two to three hours a day manually scanning 20 websites each day? An aggregator completes this process in seconds. In total, you could save up to 60-90 hours of time a month. At a cost of $50 per hour (for an analyst), that will yield you between $3,000-$4,500 of savings.

No Human Factor. An aggregator will never get tired or get distracted when processing information. For example, if there is a major model update at 3 AM, the aggregator will not miss it. As the cryptocurrency market operates 24/7, the faster information is consumed, the more profits will be created by participating in it.

For more information check out the ASCN project. A great example is Falcon Finance. Their workforce consisted of three analysts who would collectively spend just under 15 hours each week keeping track of 50+ different AI and blockchain projects. After implementing an automated scraping process that used markdown exports, their total amount of time spent on all the data was reduced from almost 15 hours to about two hours.

Collection, filtering, formatting, sorting, and categorizing – it can all be automated.

Three ways to scrape news from an AI source

There are three different ways to scrape news from an AI source: HTML Parsing, RSS feeds, and Hybrid approach. Below is a quick overview of each option:

1. HTML Parsing: This is the most universal option available. A script is created that loads a page and analyses the whole document object model (DOM) by using the properties of tags, classes, and attributes to find the information needed. Since not all of these website offer RSS feeds or APIs, it is an option to use parsing. However, because many of these pages change their layout often and without notice, if a page's layout changes and the parser is not modified, the scraper will break.

2. RSS Feeds: RSS Feeds are an official way to get the latest headlines, links, and dates in an XML format. Using an RSS Feed has the following advantages: they are always supported; they allow for server-side filtering; and are relatively stable in terms of longevity. The disadvantages are: many required that the user register with the website to generate an API key; there are limits to the number of requests that can be made to an RSS Feed.

3. Hybrid Approach: Typically, 68% of businesses using a hybrid method for collecting news utilize APIs for their main source of news, RSS for stable news, and HTML parsing for niche blogging sites without an API. 

Summary of Multiple News Sources

In order to create a full picture, information from only one source does not provide an accurate reflection of the industry; you require input from multiple parties. Aggregators have the primary role of compiling various sources together into one two-dimensional feed dedicated to Artificial Intelligence news. Good sources for monitoring AI include:

• Top Technology Publishers: TechCrunch (start-ups and investing), The Verge (new products), Ars Technica (research and analytical content), VentureBeat (business-oriented).

• Specific Publications Concentrating on AI: MIT Technology Review, AI News, and Google's and OpenAI's blogs.

• Places to Find Scientific Research Material: arXiv.org, Papers with Code, Hugging Face Papers.

• Social Networks to Provide Information: Twitter/X (hash tags for #AI, #MachineLearning), Reddit (sub-reddit for r/MachineLearning, r/artificial), LinkedIn.

From a technical perspective, aggregators contain a list of end points (RSS feeds, APIs, HTML pages) where the aggregator conducts its polling of the source information, each with its own polling frequency. Also, the aggregator must include deduplication to avoid having duplicate messages recorded for multiple sources when the same information is in duplicate form from different sources at the same time, leaving behind only the one message with the other sources referenced in the message.

Currently, at ASCN.AI, there are 30+ sources with polling every 10 minutes. This allows clients access to updated analytics and places them in advance of the market, sometimes by hours or days.

How Many Times Is Content Updated and How Is It Filtered?

The number of times that a source is polled for new information is based on the significance of the news and how quickly the information changes:

• 7 to 15 minutes - Very significant sources like the official OpenAI blog, the official Google AI blog, or Product Hunt.

• 30 minutes to 2 hours - Major news sites such as TechCrunch and The Verge.

• Once daily - Academic research repositories and thematic-based blogs.

By applying filters to previously explained information, the amount of noise to reduce in the above bullet points is achieved through the following methods:

• Keyword search: "GPT," "LLM," "funding," "Series A," amongst many others tools.

• Source filtering: There is a whitelist of reputable publications and a blacklist for click bait.

• Deduplication: The similarity of the text portions is calculated (Levenshtein distance, TF-IDF) and all portions of the text have been cut out that are duplicated. Use of sentiment analysis is one of the ways to assess if content is positive, negative or neutral in tone.

Automated data aggregation uses relevant filtering to reduce the noise of incoming data by 75-80% and achieve an accuracy of 85-90% in selection.

Data Formats: Why use Markdown?

Markdown is simply a text format with basic headings, lists and links that can all be easily scanned and read in raw form (no rendering required). Creating and reviewing data within a Markdown document provides users with quick, concise information about a given topic.

• Universality. Markdown is supported by many popular systems, including Obsidian, Notion, Roam Research, Logseq, GitHub and others. By using Markdown, you can easily place news articles into your own knowledge management system (KMS) from any source.

• Automation simplicity. Creating and managing content within Markdown format is a matter of joining strings together, so there is no need for cumbersome parsing or complex formats.

• Integration with your PKM system. The use of tags and internal linking within Markdown creates a web of interrelated content that makes it easier to search and cite when needed.

Example of AI News Formatting in Markdown

# AI News Digest — 2024-01-15
## OpenAI Unveils GPT-4.5 Capable of Multimodal Interaction
**Source:** [OpenAI Blog](https://openai.com/blog/gpt-4-5-release)  
**Date:** 2024-01-15 09:30 UTC  
**Tags:** #GPT #OpenAI #Multimodal
OpenAI has introduced GPT-4.5 with multimodal support. It is an advanced language model with support for images, audio, and video. The new version shows a 15% improvement in performance on the MMLU and HumanEval benchmarks.
**Key Features:**
- Native image processing (without CLIP)
- Video support up to 5 minutes
- 20% reduction in API costs
[Read more](https://openai.com/blog/gpt-4-5-release)
---
## Google DeepMind has introduced Gemini Ultra for Corporate Sector
**Source:** [TechCrunch](https://techcrunch.com/2024/01/15/google-gemini-ultra-enterprise)  
**Date:** 15 января 2024 года в 11:00 по Гринвичу  
**Tags:** #Google #Gemini #Enterprise
Google DeepMind has launched an enterprise version of Gemini Ultra with private cloud deployment support. It is designed for analyzing large volumes of corporate documents.
[Continue reading](https://techcrunch.com/2024/01/15/google-gemini-ultra-enterprise)

Comparison of Popular Formats

The processing of digests at ASCN.AI demonstrates this: All news are created in Markdown and then imported automatically into an Obsidian vault with tags and links. Both analysts and traders can count on having access to a well-organised, easy-to-review archive of their digests.

Delivering News Automatically via AI and Newsletter

News aggregation is only the first step; the next step is to distribute the news to your audience at a time and in a way that is most convenient for them. Automation can change an aggregator into a media tool rather than just a way to collect data.

An example of how automation works:

1. Data will be collected - An aggregator will poll sources and only save news to databases (SQLite, PostgreSQL, Google Sheets)

2. Create a digest - To create a digest, each day a script will find and select all of the news items over the last 24 hours and create a Markdown or HTML file that is sorted by how important they were (i.e., number of mentions, number of reposts, source rating for that source).

3. Personalization — By using subscriber segments (investors vs developers vs businesses), a segment will receive items that are only of interest to them.

4. Delivery - Via email services (SendGrid, Mailchimp, ConvertKit), Telegram, Slack, or on a blog.

5. Analytics - We track open rates, click-through rates, and popularity of topics to determine how to improve content quality.

Statistically, an automated email campaign has a 119% increase in open rate and a 152% increase in click-through rate due to the ability to personalize messages, and timely delivery of messages.

Popular Email Automation Tools

• SendGrid - an API for developers with limited usage (100 emails/day), paid plans starting at $15/month for up to 40,000 emails with full control over content.

• Mailchimp - No code tool with Visual Editing and Automated Workflows with Free plan for up to 500 subscribers, paid plans starting at $13/month.

• ConvertKit - developed for content creators with the ability to automate newsletter series and plans start at $29/month for 1,000 subscribers.

• N8N - An open-source platform designed to help people build complex workflow automation. It is free to download and use if you have a self-hosted solution.

• ASCN.AI NoCode - A no-code platform for ready-made modules for filtering, scraping, formatting, and sending news to Telegram/email, which can be set up in 10 minutes starting at $29/month.

Automating Newsletters through Scraping Case Studies

Example #1: Weekly AI Digest for Investors
A venture capital fund sought timely updates on funding rounds for AI startups without any delays.
Solution: By utilizing a combination of TechCrunch, VentureBeat, and Crunchbase and conducting real-time searches based on keywords such as "Series A", "Funding Round", and "AI Startup" the aggregator will periodically scrape these sources. Each week (Friday) a Markdown report will be compiled containing a table of the AI startups, amount, funding round, and investors for each and the report will also be posted in Notion and sent to partners.
Results: The venture fund has access to varieties of data sooner than its competitors; therefore, the fund was able to achieve 3 investments in 1 year for a 4x return based on this overall process.

Example #2: Customized Newsletter for ML Developers
A tech startup maintains an ongoing record of updated release information for models, datasets, and libraries that are available to developers and the marketplace.
Solution: The aggregator will perform periodic searches on Hugging Face, Papers with Code and Trending on GitHub using tags to determine what the best 5 new models would be and send that information to the team each morning via Slack.
Results: The development team saves an average of (1.5) hours daily and the average acceleration is approximately 30% faster training models.

Example #3: Competitor Monitoring
An AI startup tracks news articles about their competitors.
Solution: Monitor Twitter, Reddit, and various news outlets to determine if any significant events occur. A digest is sent to the marketing department every 12 hours via Telegram outlining notable mentions of their competitor.
Results: The ability to respond quickly to their competitor's actions allowed the startup to have a successful PR campaign regarding their competitor's price change.

How to Set Up AI News Scrapers

An overview of popular AI news scraping tools is an excellent reference for news professionals. There are many different options available to automate news scraping from websites. The following lists show which tools have been designed for specific purposes (i.e., Automation, Development, or Utilization), their complexity (Easy, Medium, or Hard), their ability to scrape data from multiple sources, whether they have a feature that allows Markdown exports, and their prices.

A brief description of the types of tools that are commonly used for automated news scraping is found below. Below that is a chart with each of the tool’s descriptions:

• Don’t ignore request rate limits. For example, if you have a maximum of ten requests and you make 100 requests in one day, you will be blocked (HTTP 429).

• Never request immediately after one another. Provide delays between requests, rotate proxies, and don't forget to check for API limits.

• Always monitor your scraping method being used, as when a site changes their layout, the parser may stop functioning. Using less prone to change selectors is a way to help monitor the success or failure of your parsing method.

• Many duplicates are not good for the reader and should be avoided at all costs. Use algorithms that compare text and URLs to eliminate duplicates before sending them out.

• Start by using 5-10 keywords as a base and using reputable sources, this allows you to capture all of the key information that you are trying to capture.

• Check for terms of use and implement official APIs, if you are doing a commercial project you should consult with an attorney regarding your legal risk.

• Create a contingency plan so in case your main source is down you will have a backup. You should use logs and notifications to track this.

• Use UTF-8 encoding when writing files; this is particularly important when you are reporting on international news.

• Please automate the entire process you should not stop at the extraction of information, but should also automate the delivery of that information from extraction to sending and archiving (using email, or Telegram).

In Conclusion

Manual searching and monitoring no longer work in an environment that is rapidly evolving and flooded with information. The true value in automated AI news aggregators is that they provide access to a continuous supply of new and related news articles in a structured format that saves you time and allows you to gain an edge over your competition.

Implementation Recommendations:

• Start small by connecting 5 to 10 of the most beneficial sources of information either via api or rss, creating basic filters and launching a normal workflow.

• In your first phase of development, use the APIs and RSS and have the HTML parsing be your secondary option if your primary api or RSS feeds become unavailable.

• Automate your entire processes from collection to delivery (email, Telegram, Slack).

• Use a single format, specifically markdown, for all content; this will simplify integration and automation processes.

• Monitor the numbers; as you will be able to measure your saved time, open rates, and overall quality of news articles; you will also be able to refine your selection criteria over time.

• For those that are just starting out, there are various no-code platforms available (i.e. ASCN.AI NoCode, n8n, or make.com).

• As you work with APIs ensure that you are abiding by the copyright and the terms of use for any site you are accessing.

• Use the capabilities of the LLMs (large language models) to analyze the trends of the articles you are collecting, the summation of those articles, and the sentiment of the articles you are collecting to analyze.

• Automating news sources is no longer an option, but it has become a requirement because we are loaded down with information. The person or organization that can collect and distribute the information quickly and accurately will win.