關於網路那些事...

Recently, have had some scheduling tasks that need to execute on the website. This will cause high effort for this manually routine job.

For reducing the effort, I build an automation mechanism to replace the routine job, here will record the process for anyone who needs refer.

This article will use an example as a case to demonstrate how to build this automation process, and the following are this example of system processing:

What is an event-driven architecture?

Traditional approaches to integrating applications involve tightly coupling them together, making changes to one system impact the other.

Event-driven architecture is an excellent way to decouples and aggregate two or more applications because it provides a flexible and scalable way to integrate and communicate between different systems sources, such as user actions, system events, or sensor data. Event-driven architectures are used to build complex and scalable systems that respond to changes in real-time.

A data lake is a centralized repository for storing and managing large amounts of raw data in various formats. Amazon Web Services (AWS) provides a comprehensive range of services that can be used to set up a data lake system. In this article, we will introduct how establish a data lake on AWS.

Recently, our payment service is building with serverless Event-Driven Architecture with SNS, SQS, and lambda function, this will support the Event callback with core payment event service (the following will show as core event service), and this core event service are integration muli-purchase 3rd vendor to one interface.

When a user has subscriptions behavior in our platform, each action send to 3rd party vendor, the 3rd party vendor will create an Event to core event service, and the core event service send the callbacks Event to project payment service’s SNS, for the final process, project service need provide a cross VPC account access SNS grants to core Event service:

3rd vendor --(callback event)--> core event service --(callback event)--> project

When developing an APP in ECS, We need to pass the environment variables to the tasks container.

Because we have sensitive and non-sensitive environment variables, so we need to take care in choosing the policy.

We have a service that sometimes got a 504 Gateway Time-Out response from ELB( actually is CLB).

When investigating the root cause, we found out that CLB’s 3 instances are normal in the current two weeks, and the 504 time-outs happened in CLB.

In this article, we will introduce the main key features of HTTP 1.0, HTTP 1.1, HTTP 2.0, and HTTP 3.0.

HTTP1.0

HTTP1.0 was published in 1996 and already obsoleted.

In HTTP1.0, each request needs to make TCP 3-way-handshake connection, which means will high cost for time-consuming and low efficiency.

In this article is a tutorial to get started with GitLab Runner, will show you how to install runner in AWS EC2 and registrations to GitLab.

Before you start, you need to make sure local with the following configuration:

• login AWS CLI
• Docker
• nodejs

AWS CLI tool - AWS SAM CLI

AWS SAM CLI is a AWS CLI tool that allows you to develop, test and analysis your application in the local environment.

In MAC environment, install SAM tool by brew:

brew tap aws/tap
brew install aws-sam-cli
sam --version

Opsworks Introduction

AWS OpsWorks is an configuration management service that provides an easy way to create and manage AWS stacks and application.

You can preset your AWS resourcce and manage the configuration, deploy those resources and monitor the status of the resources activity.

Here is a AWS Cloudformation IaC process, it need you manual setting:

Code templates (Yaml or Json) -> Upload to S3 bucket -> Create stack (can created by console, api or CLI) -> Stacks and resource provisioned.

This is part II to illustrate how to setting secret to kubernetes pod local file.

(Aobut the setting secret to environment variables can reference : How to setting kubernetes secret to environment variables )

In Kubernetes, some sensitive information needs to be managed in Pods, which can be managed by Kubernetes secret.

Here we will demonstrate how to manage secrets and save to pod local file.

In Kubernetes, some sensitive information needs to be managed in Pods, which can be managed by Kubernetes secret.

Here we will demonstrate how to manage secrets.

This article will illustrate how to run a Kubernetes in mac M1 local, and learn how to use kubectl command to deploy a service. More detailed concept can reference Kubernetes documents

In this article, we will create a simple golang application CI/CD process and push it to Docker hub by github action. Finally, will illustrate an example of how to use the terraform and docker-compose to deploy a local docker environment.

Create Docker hub repository

First, create a repository on Docker hub, here we create a repo name adon988/go-github-action-helloworld

Generate Github Actions workflow

Create a project folder .github/workflows/ and add github actions yaml file

Here we used the gorilla module to build a websocket application, and this tutorial will focus on building up a websoccket server.

About client part will implement with the chrome extension: Websocket King.

Get gorilla module

go get github.com/gorilla/websocket

Build a websocket server

Following are examples of simple websocket servers built with gin and gorilla,

Gin creates a get request and upgrades this HTTP connection to websocket connection by upgrader.Upgrade.

Golang: 解決 Go Websocket upgrade:websocket: request origin not allowed by Upgrader.CheckOrigin 跨域問題

錯誤訊息：

upgrade:websocket: request origin not allowed by Upgrader.CheckOrigin

原因： 如內文所述，在 Go Websocket 判別跨域來源不合法

解決方式： 需要在 websocket.Upgrader 設定 CheckOrigin 來源，例如下方允許所有來源

var upgrader = websocket.Upgrader{
	CheckOrigin: func(r *http.Request) bool {
		return true
	},
} // use default options

例如，在使用 Chrome Websocket King Client 擴充時，可以只允許這個來源，例如：

var upgrader = websocket.Upgrader{
	CheckOrigin: func(r *http.Request) bool {
		origin := r.Header.Get("Origin")
		return origin == "chrome-extension://cbcbkhdmedgianpaifchdaddpnmgnknn"
	},
}

AWS Copilot CLI 是一個開源的 CLI ，可以方便地用於管理 AWS App Runner, AWS ECS, AWS Farget.

只要透過簡單的指令就可搭建起 containerized application，

在本文會先介紹 ECS 以及如何透過 Copilot 來部署應用到 ECS 。

AWS ECS 簡介

AWS ECS(Elastic Container Service) 是一個簡單且高擴充的 container cluster 管理服務， 可以透過 task definnition 定義 container 並且直接運行在 serverless infrastructure (Fargate)，或者可以選用 EC2 來做更多的調控。

下方是一個 ECS Object 示意

在開始創建 ECS ，有以下流程

Container definition : 會先需要選擇 image 以及規格 （cpu, RAM….)

Task definition : 用於定義 APP 的藍圖，可以用來描述要部署的一個或多個 containers (最多10 個) ，透過 attributes 設定 container 配置(network, cpu, ram..) 等

這邊列出幾個常見的 shards & replicas 問題及流程機制：

本文紀錄 Linux 安裝及配置 Elasticsearch 單節點以及 cluster 流程：

這裡記錄一些 Elasticsearch 常用的索引操作方式，方便之後查詢參考