Introduction​

GitHub Actions is the official CI/CD workflow service provided by GitHub. It is aimed at making it easy for open-source project contributors to manage operational maintenance, and enable open-source communities to embrace cloud-native DevOps. GitHub Actions is integrated into most of my open-source projects including Crawlab and ArtiPub. As a contributor, I think GitHub Actions is not only easy to use, but also free (which is the most important). Therefore, I hope this article will allow open-source project contributors who are not familiar with GitHub Actions, to really get ideas on how to utilize it and make an impact.

Starting from documentation​

For those who are not familiar with GitHub Actions, it is strongly recommended that you read the official documentation first, where you can find Introduction Video, Quick Start, Examples, concepts, how it works, etc. If you read through the docs, you can easily do GitHub DevOps with your own experience in CI/CD. References of all codes in this article can be found on the official documentation,

Ideas​

Let's first figure out what we would like to implement, i.e. using GitHub Actions to run a web crawler to get daily ranking from GitHub Trending.

Introduction​

"No code is the best way to write secure and reliable applications."--Kelsey Hightower

Many developers have probably more or less heard of Unit Tests, and even written one and are familiar with it. However, in the volatile and fast changing environment, unit tests seem to be in an embarrassed situation. Developers know it is useful, but treat it with neglect. "The schedule is tight. What time do we have for unit tests?" Does it sound familiar?

What is Unit Test?​

Unit Test is some lines of code written by developers to validate whether their own functional codes can run as expected. If the code is not passed, it means the functional codes are problematic.

This self-testing method looks self-deceiving, similar to taking an exam with official answers. In validation area, this term is called White Box Test. The counterpart of White Box Test is Black Box Test which uses other methods to validate things. Unit Test is White Box Test, and higher-level testing methods such as Integration Test, End to End Test, and UI Test are all Black Box Tests. Unit Tests only test code itself.

What are the benefits of unit testing?​

Unit testing is a very useful tool in Agile Development. Some agile frameworks, such as eXtreme Programming (XP), requires that every feature must be covered by unit test cases. My previous article Talking Agile: Are you sure your team is practicing Agile properly mentioned the importance of unit tests.

Introduction​

"According to incomplete statistics, the proportion of white hair of data workers is higher than the average of the same age group." by a data worker

Data, a familiar but mysterious word, has become a totem pursued by everyone. Managers love fancy data reports, data analysts are keen on building complicated statistical models, and salesmen take dashboards as compasses to see whether they can complete their KPIs. Since over ten years ago, the data industry has been developing fast, and there have been some novel yet formidable jargons, such as Big Data, Data Science, Data Lake, Data Mesh, Data Governance. Yet the "traditional" terms are still abstruse: Data Warehouse, Business Intelligence, Data Mart, Data Mining. What is more headachy is that many people are still unable to understand their relationship with recently popular concepts such as Artificial Intelligence, Machine Learning, and Deep Learning. These hot buzzwords are the results of aggressive development in the data area.

Professional Doctor or Fortune Teller?​

Years ago, with the rapid development of the Internet industry, the bubble of the data industry was getting larger. Data, the by-product of the Internet applications, has large volumes and diversities. Data owners would like to get the most out of it and regard it as the gold mine. Therefore, data mining engineers became one of the most popular professionals. Later, a brand new yet more popular position Data Scientist emerged as "the sexiest job in the 21st century".

The popularity of data scientists is its requirement for abilities and experience in various areas:

• Programming Skills: at least able to use Python or R to do data cleansing, analysis and modeling.
• Mathematics and Statistics: familiar with probability theory, calculus, and discrete mathematics.
• Business Knowledge: deep understanding of market, process and macro trends in related areas.
• Communication Skills: able to convey insights and analysis results in a human-friendly way.

Introduction​

Task Scheduling is one of the most important features in software systems, which literally means assigning and executing long tasks or scripts according to certain specifications. In the web crawler management platform Crawlab, task scheduling serves as a core module, which you may wonder how to build it from scratch. This article will introduce you how to build a simple but useful task scheduler with Go.

Idea​

Let's focus on what we need for the task scheduling system.

• User Interface: API
• Scheduler: Cron
• Execute Tasks: Executor

Below is the basic process.

We can use HTTP API to create scheduled tasks, and the executor will execute scripts periodically based on their specifications.

Introduction​

"You'd better stay late tonight. Our boss is angry about the progress." -- An ambitious but worrying project manager

Agile, a word once standing for agility and speed, has become one of the sexiest but seemingly abstruse jargons often mentioned by software developers in the IT industry over past 20 years. Agile Development is the primary option of project management methodologies for many teams, because it is simple, agile (literally), and more importantly implying fast. Think about countless programming labors who are willing to work overtime voluntarily because of hard deadlines.

Agile = Deliver Fast?​

When your dev team suffers deferred releases, product quality degradation, poor team moral and fading customer relationship, your friends probably will recommend you using agile development, "Hey, buddy. I heard that agile is pretty awesome. You should probably take a shot." However, after your team has been practicing agile for a while, you are very likely to find it not good enough: loads of bugs after production releases, increasing number of overtime hours, and endless new requirements each of which is the highest priority. You are trembling with fear in the regular meeting with your boss who may ask, "I heard that you guys are improving our capability of delivering with Agile. This is great! So can those important features be available next week?"

Agile is the same as delivering fast, this is one of the biggest myths from many agile practitioners. Literally, Agile doesn't necessarily mean fast; instead, introduction of Agile will lead to more work and therefore lower delivering speed. Yeah, you are right. Technically speaking, Agile will reduce the speed of delivery. For example, Extreme Programming (XP, one of the most popular agile frameworks) requires unit test cases to cover all features and functions, which would result in 1-2 times of extra code.

So, why do people claim that Agile works great in software development? As it can neither increase development efficiency, nor does it reduce workload, what on earth is this thing useful? This is a typical and the most critical question for Agile practitioners. If you cannot provide timely answers that is satisfied to the team, they will ultimately drop Agile and head back to their old way.

Preface​

Today's heroes in this world are only you and me. -- Cao Cao, "Romance of the Three Kingdoms"

For programmers who have been working in the IT industry for many years, if asked about the most mainstream backend programming language in China, I believe most would say Java. This isn't surprising, as Java has existed for over 30 years, has a massive user base and ecosystem, and seems to hold an absolute dominant position in the software engineering field. However, as ancient wisdom says: "Those who win the hearts of the people win the world." The programming language with the most users isn't necessarily the most beloved by developers. According to StackOverflow's 2021 survey of 82,914 developers on programming language satisfaction, only 47% liked Java, ranking it beyond 20th place, barely higher than PHP, C, and COBOL. On the other hand, we can see from the survey results that C#, often called the "knockoff Java," actually achieved 62% satisfaction among developers, 15% higher than Java. Although C#'s satisfaction still has a considerable gap compared to Rust and TypeScript, we can see that C# as an alternative programming language to Java is gradually gaining advantages in development efficiency, deployment convenience, and documentation completeness. Due to work requirements, I've used both C# and Java to develop numerous projects, giving me some understanding of their similarities, differences, advantages, and disadvantages. I believe there are good reasons why C# is more popular among developers than Java - it offers a great development experience.

Due to length constraints, the complete introduction to C# principles and practical applications (i.e., why I recommend using C# to build large-scale backend applications) will be split into a series of articles. This series will deeply analyze this "young" programming language from dimensions such as syntax features, development patterns, ecosystem, deployment and building, using the cross-platform framework .NET Core as an example to introduce how to build large-scale backend applications with C#.

This article is the first in the C# series, primarily introducing some modern syntax features of C# and how they improve development efficiency.

Introduction​

A bad plan is better than no plan.

坏计划也好过没有计划。--彼得·蒂尔《从0到1》

In software development engineering, there are rarely lone wolf programmers. This is because modern, commonly seen software projects are usually very complex, requiring substantial human resources, resources, and time. Having a single developer complete a large software project alone would be like "an old man moving mountains." Therefore, software development is inseparable from team collaboration and project management. Project Management, simply put, is a methodology for orderly organizing, planning, executing, and completing various tasks in a project. Of course, the actual scope of project management goes far beyond this, usually involving resource allocation, priority setting, progress tracking, etc. It's a product of the Industrial Revolution and a branch of modern management science that can significantly improve engineering completion efficiency and success rates. This article mainly discusses software project management, which is very different from traditional project management in construction engineering, mechanical engineering, etc. Early IT project management borrowed from traditional project management methodologies like construction engineering, playing an important role in the early information age and significantly improving software development and collaboration efficiency. However, with the rapid development of the IT industry, consumer product demands change rapidly, and market conditions have become increasingly volatile. Traditional software project management models can no longer meet software development needs. Therefore, modern software development models, such as Agile Development, emerged and became the preferred choice for many internet companies.

What are the drawbacks of traditional project management models (such as waterfall)? What improvements do modern project management models (such as agile) offer? Should we completely abandon waterfall models and fully embrace agile development? As a programmer, should you master some project management knowledge and related tools? As a team leader, how should you establish project management processes to ensure development efficiency and quality? If readers have similar questions, this article will provide detailed analysis and answers.

Introduction​

Those outside the city want to get in, those inside the city want to get out. -- Qian Zhongshu "Fortress Besieged"

With the continuous popularity of Container Orchestration, Microservices, Cloud Technology and other trends in the IT industry, Golang (Go language, abbreviated as Go), born at Google in 2009, is increasingly welcomed and sought after by software engineers, becoming today's hottest backend programming language. In the list of software projects developed with Golang, there are star-level products like Docker (container technology) and Kubernetes (container orchestration) that have disrupted the entire IT industry, as well as powerful and practical well-known projects like Prometheus (monitoring system), Etcd (distributed storage), and InfluxDB (time-series database). Of course, Go language's application domains are by no means limited to containers and distributed systems. Today, many large internet companies are extensively using Golang to build backend Web applications, such as Toutiao, JD.com, Qiniu Cloud, etc. The web scraping field, long dominated by Python, is also being continuously challenged by Golang due to the rise of the simple and easy-to-use scraping framework Colly. Golang has become the programming language that most software engineers want to learn today. The image below shows relevant results from HackerRank's 2020 programmer skills survey.

So, is Go language really a lifesaver for backend developers? Can it effectively improve programmers' technical capabilities and development efficiency, thus helping them advance further in their careers? Is Go language really worth spending a lot of time learning in depth? This article will provide detailed introduction to Golang's language characteristics, its advantages and disadvantages, and applicable scenarios. With the above questions in mind, it will analyze various aspects of Go language to help programmers new to the IT industry and developers interested in Go to further understand this popular language.

Preface​

The most incomprehensible thing about the world is that it is comprehensible.

The world's most incomprehensible aspect is that it is comprehensible. -- Albert Einstein

Open-source has created today's thriving software industry. Open-source enables developers worldwide to collaborate on excellent utility projects, also known as "wheels," benefiting companies and individuals of all sizes while showcasing creators' and contributors' technical prowess. Many developers today extensively use open-source projects as third-party libraries or dependencies to complete development tasks faster and more efficiently.

I'm no exception. Recently, while refactoring the Crawlab frontend using Vue 3, I used the upgraded version of ElementUI developed by the Element team - the new UI framework Element Plus rebuilt for Vue 3. The Element team completely refactored Element Plus using Vue 3, fully embracing TypeScript. Compared to the previous Vue 2 version, it enriched some components while maintaining consistent styling and usage patterns. Some APIs even became more streamlined. Therefore, during the initial refactoring phase of Crawlab's frontend, I didn't encounter major obstacles. Combined with previous development experience, the development process felt smooth and familiar. However, the good times didn't last long. As the project continued developing, I encountered some technical difficulties. More precisely, I faced limitations from the Element Plus framework itself when implementing complex features. Although I eventually found ways to solve the problems, I deeply experienced the difficulties of hacking open-source project source code. Therefore, I'd like to take this opportunity to share my experience in mastering open-source code with readers.

This article will begin with my experience solving Element Plus problems, progressively discussing issues with open-source projects or frameworks, and further discussing methods and techniques for mastering open-source project source code, sharing thoughts on reading, understanding, and modifying source code. This article primarily focuses on methodology discussions without too many technical details, making it accessible to readers of any professional background.

Introduction​

The only constant in the world is change.

世界上唯一不变的是变化。--《谁动了我的奶酪》作者 斯宾塞·约翰逊

The IT industry changes too fast, especially frontend development. If you could travel back 10 years and meet a Web development software engineer, they would definitely tell you that mastering frontend means being proficient with jQuery and solving IE browser compatibility issues. However, with the continuous development of frontend, jQuery faced "official retirement" and gradually exited the historical stage (element selection and manipulation were unified by standard DOM APIs); the much-criticized IE browser compatibility problems, due to IE's gradually shrinking market and the emergence of compatibility tools (Polyfill), have been downgraded from core optimization issues to minor annoyances, no longer being standard equipment for frontend engineers.

Today's frontend development has a dazzling array of professional terminology and complex technology ecosystems, which may make engineers new to frontend development feel anxious: there's simply too much to learn. Modern frontend engineers who don't understand Webpack, Babel, Node.js, NPM/Yarn, ES6/7, React/Vue, Sass/Less, TypeScript, ESLint, Canvas/SVG and other modern frontend knowledge will find it difficult to convince others of their professional background. Frontend engineers in 2021 may be true engineers in every sense of the word. They typically need to apply extensive professional knowledge to solve engineering problems, including how to modularize projects, how to design interactions between components, how to improve reusability, how to enhance bundling efficiency, optimize browser rendering performance, etc. They no longer just need the HTML/CSS/JS routine to develop static pages like before.

This article will focus on the theme of modern frontend development to introduce various important technologies of frontend engineering in detail, helping readers understand how the complexity and diversity of modern frontend pages are constructed. This article is a popular science piece about frontend engineering—even if you don't understand frontend technology, you can benefit from this article.