LegiNote Project Development Story 2 - Tech Stack and Workers

Hello, this is StatPan.

I'm writing about the development process of the LegiNote side project.

Please refer to the following link for the previous episode.

Part 1

Basic Architecture

Leginote is planned to be developed as a web service first, and we intend to use the following technology stack.

Frontend: Svelte

Backend: Rust (with a strong consideration for rupring, but this may change)

Worker: Go (aiming for extreme std usage excluding essential dependencies)

Data Analysis: Python (from preprocessing to future AI integration)

Among these stacks, I plan to focus on the worker and data analysis parts, which are the areas where I'm primarily contributing, and serialize the development process in this blog series.

Let's start with the worker.

Worker

Language Selection

I started programming with Python, and since my current main work is in the data science field, Python is the language I'm most familiar with. However, having only known scripting languages, I developed a desire for a robust compiled language, which led me to become interested in Go and Rust.

Especially recently, Rust was the language I've been following with the most interest. When Rust is mentioned, C++ and Golang are often mentioned together. (While C++ is also an excellent language, I excluded it for network-related service development in this project. I'm not sure if I'll directly use it in the future unless I'm dealing with the internals of a deep learning framework.)

Through learning Rust and Golang and using them for simple network requests, I briefly identified their strengths and weaknesses. It would make this post too long to delve into the pros and cons of both, so I'll keep it brief here and provide a separate explanation in a different post. In short,

In the network field, for a project with an uncertain future regarding revenue generation and the ability to configure it without additional dependencies,Golangwas chosen.

Worker Role

The worker's role is to collect and update data periodically through OpenAPI sites. This can be divided into three parts:

1. Setting up the collection logic for specific information

2. Setting up a runner to execute the collection logic

3. Loading the collected data into the DB

The goal is to apply and extend these three steps identically to various API data.

Project Structure

Various architectures exist for project structure, and using a framework can help you start with a structured project. However, as mentioned earlier, this project aims to minimize dependencies, so we decided to structure the project based on the defined logic, separating folders for each function.

The project will be structured as follows:

leginote-worker-bill/
├── main.go
├── api/
│ └── client.go
├── db/
│ ├── connection.go
│ └── repository.go
├── util/
│ ├── error.go
│ └── runner.go

│ └── config.go
├── worker/
│ └── worker.go
├── go.mod
└── go.sum

api - Data collection is done through public data OpenAPI. Therefore, the basic code for communicating with OpenAPI will be placed under the api folder. Currently, only one API is used as the main API, so it is structured with only one client file.

db - The logic for communicating with the DB is defined here. The connection code in connection only handles the connection, and the repository contains logic for executing various upsert SQL statements. Having the SQL itself in the code is too complex, so I plan to move the SQL to separate files later.

util - This is where various logic that assists functions during development is defined. error is a temporary script created for convenient handling of error results from Go. As you'll see in later development stories, this becomes a huge source of the butterfly effect (?).

The config file is a file that defines a function to read the .env file directly, rather than using a separate package when reading the .env file.

worker - This is where the main logic for using the api and db logic appropriately to achieve the desired behavior is contained. When the number of workers increases, this part will need to be expanded accordingly. For now, only one main worker operation is included.

In the future, I plan to continue developing the code while maintaining the principles of layered architecture.

Project Dependencies

pq for PostgreSQL usage

sqlx for query usage

Worker Conclusion

It might be a bit boring to just list theories without code, and I feel like I'm getting a little carried away, so I'm wrapping things up a bit quickly.

This concludes Part 2 already. I hope that from Part 3 onwards, we can include at least a line of code and make the discussion more engaging.

During the LegiNote development process, if you have any features you think are necessary or any questions about the development, feel free to leave a comment!