Introduction to SRead​

SRead is a smart reading assistant, whether you enjoy reading articles or viewing electronic papers, you can utilize SRead for assisted reading. SRead supports intelligent summarization, capable of extracting key information from the reading material and summarizing it; additionally, it can perform intelligent Q&A, answering any relevant information within the article. Moreover, SRead's mind map feature can help readers quickly grasp the outline of the entire piece.

Chrome Extension​

The new Chrome extension of SRead brings a major upgrade to the browser reading experience. Once this extension is installed, users can directly enjoy all the features of SRead on Chrome browser without the need to download any additional applications. This extension includes a simplified toolbar, making it easy for users to quickly access the intelligent summarization, intelligent Q&A, and mind-mapping features while reading. Another important feature of this extension is that it can automatically recognize web page content, providing real-time intelligent assistance to users, making the reading experience smoother and more efficient.

Installation and Usage​

Installing the SRead Chrome extension is very straightforward. Users first need to log on to the SRead website (https://sread.ai), and register/login with Gmail or WeChat. Then visit the Chrome Web Store, search for "SRead", and click the "Add to Chrome" button. Once the installation is complete, the SRead icon will appear on the toolbar, clicking the icon activates the extension and users can start using it.

Introduction​

While exploring the field of artificial intelligence, we often need to leverage existing APIs to implement specific functionalities. Recently, while exploring the Langchain library, I discovered an interesting feature: using OpenAI's function call API to perform specific operations in a chain. This not only demonstrates how to obtain structured outputs from ChatOpenAI but also how to create and execute function chains. This feature offers us a new possibility, enabling the execution of multiple functions within a chain. Through this approach, we can obtain structured outputs based on specific inputs, thus providing more accurate data for subsequent operations.

LangChain OpenAI Functions​

Firstly, we need to understand how to obtain structured outputs from ChatOpenAI. In the Langchain library, there's a create_structured_output_chain function that can accept either a Pydantic class or JsonSchema for structured output formatting. This way, we can force the model to return outputs in a specific structure, facilitating subsequent processing.

For instance, we can create a Person class to describe basic information about an individual:

from langchain.pydantic_v1 import BaseModel, Field

Introduction​

Knowledge Question-Answering Systems (KQA) are one of the core technologies in the field of Natural Language Processing (NLP). They help users to quickly and accurately retrieve the information they need from a vast amount of data. KQAs have become essential tools for individuals and businesses to acquire, filter, and process information. They play a significant role in various domains like online customer service, smart assistants, data analytics, and decision support.

Langchain not only offers the essential modules to build a basic Q&A system but also supports more complex and advanced questioning scenarios. For example, it can handle structured data and code, allowing Q&A operations on databases or code repositories. This significantly expands the scope of KQA, making it adaptable to more complex real-world needs. This article will introduce how to build a basic KQA system with Langchain through a simple hands-on example.

Hands-On​

Next, we will go through a hands-on example to guide you through building a KQA system with Langchain.

1. Document Loading and Preprocessing​

Introduction​

In the modern data-driven era, acquiring and managing online information has become crucial. To provide powerful support for enterprises and developers, Crawlab has emerged as an enterprise-level web scraping management platform characterized by being ready-to-use out of the box. Regardless of your team size, Crawlab can provide professional and efficient web scraping management solutions.

Core Features​

Crawlab's core features include distributed system management, spider task management and scheduling, file editing, message notifications, dependency management, Git integration, and performance monitoring, among others. Its distributed node management allows spider programs to run efficiently across multiple servers. No more worrying about manual uploading, monitoring, and deployment hassles - Crawlab automates all of this, ensuring you can easily schedule spider tasks and view spider program running status and task logs in real-time.

Key Highlights​

"Reading is an adventure of the mind, and knowledge is the fuel for the soul."

In today's information explosion, reading has become an indispensable part of each of us. However, traditional reading methods often drown us in a sea of information, making it hard to discern what knowledge is genuinely useful. Against this backdrop, SRead comes into being.

What is SRead?​

SRead is an AI-based reading assistant specially designed to enhance your reading experience. It is not just an e-book reader but also your personal reading advisor and assistant.

Intelligent Q&A: Answers On-Demand​

No longer need to search online or refer to other materials when interrupted during reading; SRead's Intelligent Q&A feature can instantly answer any questions related to the content or topic at hand.

Introduction​

Nowadays, generative AI applications are emerging like mushrooms after rain, overwhelming in their abundance. Large Language Models (LLMs) have become exceptionally popular with the release of ChatGPT and are a typical example of generative AI applications. However, LLMs have flaws. One significant problem is hallucination: for unfamiliar questions, LLMs fabricate answers that appear professional but have no factual basis. To solve this problem, many AI-based knowledge Q&A systems adopt Retrieval-Augmented Generation (RAG) technology, enabling LLMs to provide fact-based answers and eliminate hallucinations. This article will briefly introduce how RAG works in knowledge Q&A systems.

LLMs​

To understand RAG, we first need to briefly understand LLMs. Actually, through extensive parameter training, LLMs can already complete many incredible NLP tasks, such as Q&A, writing, translation, code understanding, etc. However, since LLM "memory" remains at the pre-training moment, there will definitely be knowledge and questions it doesn't know. For example, ChatGPT developed by OpenAI cannot answer questions after September 2021. Additionally, due to the existence of hallucinations, LLMs appear very imaginative but lack factual basis. Therefore, we can compare LLMs to knowledgeable and versatile sages who can do many things but have amnesia, with memories only staying before a certain time and unable to form new memories.

To help this sage achieve high scores in modern exams, what should we do? The answer is RAG.

Introduction​

Kaggle is an online community and data science competition platform for data scientists, machine learning engineers, and data analysts, featuring many rewarded data science competitions and datasets. The Kaggle community is very famous in the data science field, with many major internet companies publishing rewarded competitions with prizes ranging from tens of thousands to millions of dollars. This article introduces a recent participation in a Kaggle beginner NLP competition, which has no cash rewards but allows learning NLP-related machine learning knowledge.

Competition Overview​

This data science competition asks participants to determine whether a tweet is about a real disaster based on a given Twitter tweet. The image below shows a particular tweet containing the keyword "ABLAZE," indicating the tweet is about a house catching fire.

Introduction​

Exploratory Data Analysis (EDA) is a task that data analysts or data scientists frequently need to complete when facing datasets. Using Python tools like Pandas and Seaborn can easily accomplish univariate analysis, bi-variate analysis, and multi-variate analysis, but using them for data exploration not only has certain technical barriers but also requires manually writing scripts for data operations and analysis. This article will introduce a very cool automated data exploration open source tool Rath, which can automatically complete EDA and become the Autopilot or Copilot of the data analysis world.

Installing Rath​

Since Rath is still in rapid iteration and its documentation isn't very complete, the fastest way to experience it is through the demo website provided on the official site.

However, if you know some frontend technology, you can still install it locally, though the steps are slightly more cumbersome.

Before starting, ensure you have Node.js 16 and Yarn installed.

Introduction​

Using web crawling programs to extract list web pages is a one of those common web data extraction tasks. For engineers to write web crawlers, how to efficiently code and generate extraction rules is quite necessary, otherwise most of the time can be wasted on writing CSS selectors and XPath data extraction rules of web crawling programs. In light of this issue, this article will introduce an example of using open source tool Webspot to automatically recognize and extract data of list web pages.

Webspot​

Webspot is an open source project aimed at automating web page data extraction. Currently, it supports recognition and crawling rules extraction of list pages and pagination. In addition, it provides a web UI interface for users to visually view the identified results, and allows developers to use APIs to obtain recognition results.

Installation of Webspot is quite easy, you can refer to the official documentation for the installation tutorial with Docker and Docker Compose. Execute the commands below to install and start Webspot.

# clone git repo
git clone https://github.com/crawlab-team/webspot

# start docker containers
docker-compose up -d

Introduction​

"If I had asked people what they wanted, they would have said faster horses" -- Henry Ford

Today is the era of artificial intelligence. Whether it is ChatGPT or the various intelligent applications that follow, many people see the upcoming sci-fi world that was almost unimaginable a few years ago. However, in the field of reptiles, artificial intelligence does not seem to be involved too much. It is true that crawlers, as an "ancient" technology, have created many technical industries such as search engines, news aggregation, and data analysis in the past 20 years, but we have not seen obvious technological breakthroughs yet: crawler engineers still mainly rely on technologies such as XPath and reverse engineering to automatically obtain web data. However, with the development of artificial intelligence and machine learning, crawler technology can theoretically achieve "automatic driving". This article will introduce the current status and possible future development direction of the so-called intelligent crawler (intelligent, automated data extraction crawler technology) from multiple perspectives.

Current Web Crawling Technology​

A web crawler is an automated program used to obtain data from the Internet or other computer networks. They usually use automated scraping techniques to automatically visit the website and collect, parse and store information on the website. This information can be structured or unstructured data.

Crawler technology in the traditional sense mainly includes the following modules or systems:

1. Network request : initiate an HTTP request to a website or web page to obtain data such as HTML;
2. Web page parsing : parse HTML to form a structured tree structure, and obtain target data through XPath or CSS Selector;
3. Data storage : store the parsed structured data, which can be in the form of a database or a file;
4. URL management : manage the URL list to be crawled and the URL list that has been crawled, such as URL resolution and request for paging or list pages.