tty console terminal
- 2020 年 07 月 22 日
- tool
历史
- 1830s年代，出现了电报，后面出现发送电报使用的teleprinter (teletypewriter, teletype or TTY)，用来发送和接手信息。
- 1950s年代，大型机出现，只有console操作面板(没有键盘、没有显示器)
图片来源 - 1960s年代，小型机出现
- 使用teletypewriter作为大型机和小型机的输入、输出设备
- 使用物理线缆连接TTY和机器的物理插口
- 给机器的系统写一个驱动器(driver，这里留意一下这个TTY driver)
- 微型机的出现，键盘、服务器的出现，不在需要TTY。操作系统在内核中，写了一个terminal emulator。不再使用TTY这样的物理设备。
- 终端模拟软件
- VT100 VT220 XTerm等
- 现在常用ssh登录到一台远程服务器上。pseudo terminal的出现
- pseudo terminal分为2部分，master部分(ptmx)，slave部分(pts/0,pts/1)
定义
console
最开始的大型机、小型机，机器上有自有的控制台 a flat board that contains the controls for a machine, piece of electrical equipment, computer etc 机器、电器设备、电脑的控制台、操作台
图片来源
tty
+----------+ Physical Line +----------+
| teletype |<--------------------->| teletype |
+----------+ +----------+
tty本身就是一种终端，teletypewriter。 一种特定的设备文件(一切皆是文件)，基本和terminal相同的意思。tty是计算机系统操作的文件,terminal是文件另一端的设备。 内核提供一些针对键盘、显示器这样的tty。另一些是伪tty,有仿真终端程序提供。
tty 子系统
+-----------------------------------------------+
| Kernel |
| +--------+ |
| +--------+ +------------+ | | | +----------------+
| | UART | | Line | | TTY |<---------->| User process A |
<------>| |<->| |<->| | | +----------------+
| | driver | | discipline | | driver |<---------->| User process B |
| +--------+ +------------+ | | | +----------------+
| +--------+ |
| |
+-----------------------------------------------+
TTY设备
+----------------+
| TTY Driver |
| |
| +-------+ | +----------------+
+------------+ | | |<---------->| User process A |
| Terminal A |<--------->| ttyS0 | | +----------------+
+------------+ | | |<---------->| User process B |
| +-------+ | +----------------+
| |
| +-------+ | +----------------+
+------------+ | | |<---------->| User process C |
| Terminal B |<--------->| ttyS1 | | +----------------+
+------------+ | | |<---------->| User process D |
| +-------+ | +----------------+
| |
+----------------+
键盘显示器连接
使用 Terminal Emulator
+-----------------------------------------+
| Kernel |
| +--------+ | +----------------+
+----------+ | +-------------------+ | tty1 |<---------->| User processes |
| Keyboard |--------->| | +--------+ | +----------------+
+----------+ | | Terminal Emulator |<->| tty2 |<---------->| User processes |
| Monitor |<---------| | +--------+ | +----------------+
+----------+ | +-------------------+ | tty3 |<---------->| User processes |
| +--------+ | +----------------+
| |
+-----------------------------------------+
SSH连接方式
+----------+ +------------+
| Keyboard |------>| |
+----------+ | Terminal |
| Monitor |<------| |
+----------+ +------------+
|
| ssh protocol
|
↓
+------------+
| |
| ssh server |--------------------------+
| | fork |
+------------+ |
| ↑ |
| | |
write | | read |
| | |
+-----|---|-------------------+ |
| | | | ↓
| ↓ | +-------+ | +-------+
| +--------+ | pts/0 |<---------->| shell |
| | | +-------+ | +-------+
| | ptmx |<->| pts/1 |<---------->| shell |
| | | +-------+ | +-------+
| +--------+ | pts/2 |<---------->| shell |
| +-------+ | +-------+
| Kernel |
+-----------------------------+
键盘显示器连接(图形界面)
通过图形界面的Terminal软件 （注意和 键盘显示连接 的区别）
- 使用pseudo terminal (ptmx & pts)
+----------+ +------------+
| Keyboard |------>| |
+----------+ | Terminal |--------------------------+
| Monitor |<------| | fork |
+----------+ +------------+ |
| ↑ |
| | |
write | | read |
| | |
+-----|---|-------------------+ |
| | | | ↓
| ↓ | +-------+ | +-------+
| +--------+ | pts/0 |<---------->| shell |
| | | +-------+ | +-------+
| | ptmx |<->| pts/1 |<---------->| shell |
| | | +-------+ | +-------+
| +--------+ | pts/2 |<---------->| shell |
| +-------+ | +-------+
| Kernel |
+-----------------------------+
pty
pseudo terminal
- PTY slave (shell(bash/zsh…))
- PTY master (terminal emulator (XTerm)) 监听键盘等输入、输出显示内容到XTerm等模拟终端
- tty driver 是master和slave的桥梁
- shell 是一个用户态的程序，可以启动子进程(如：cat)来和内核交互。
links
Milvus
- 2020 年 07 月 19 日
- search
What is Milvus
- vector similarity search engine, 向量相似度搜索引擎
- vector index libraries
- Faiss
- NMSLIB
- Annoy
工程能力
- 分片 / cluster sharding
- 读写分离 / read/write separation
- 水平扩展 / horizontal scalability
- 动态扩展 / dynamic scalability
- on docker/k8s
场景
- 图片、视频、音频
- 文字搜索、推荐系统、交互式问答系统和其他文字搜索的领域
NN-Nearest Neighbor
- K-NN
- R-NN (Radius Nearest Neighbor)
- ANN (Approximate Nearest Neighbor)
向量检索方法
如何找到相关的doc(召回，把所有的doc都召回，也是一种方法）
- BF 暴力 Brute Force。这个就是把所有召回，适合小数据量
- KD-Tree 适合维度比较小的场景
- HC 分层聚类 Hierarchical Clustering
- HNSW 分层的可导航小世界 Hierarchical Navigable Small World
- PQ 乘积量化 Product Quantization
- KNN 近邻图 Graph
- QGraph 量化图 Quantization Graph
- LSH 局部敏感哈希 Locality sensitive hashing
相关项目
- search time
- search quality
- memory used per index vector
- training time
- need for external data for unsupervised training nmslib Nearest neighbor search-NNS
rfc2616 http
- 2020 年 07 月 04 日
- protocal
3.4 字符集 3.5 Content Codings 3.6 Transfer Codings
4 HTTP消息
4.1 消息类型
- Request : client to server
- Response : server to client
- 消息数据
generic-message = start-line //start-line肯定有
*(message-header CRLF) //0到多个 header
CRLF // 空行
[ message-body ] //这是消息体(message-body),注意和entity-body 区分
start-line = Request-Line | Status-Line // 请求行 | 状态行
4.2 消息头
- 普通的头
- 请求头
- 响应头
- entity头(fields)
4.3 消息体
message-body = entity-body
| <entity-body encoded as per Transfer-Encoding>
4.4 消息长度
4.5 通用头
general-header = Cache-Control ; Section 14.9
| Connection ; Section 14.10
| Date ; Section 14.18
| Pragma ; Section 14.32
| Trailer ; Section 14.40
| Transfer-Encoding ; Section 14.41
| Upgrade ; Section 14.42
| Via ; Section 14.45
| Warning ; Section 14.46
5 Request 请求
Request = Request-Line ; Section 5.1
*(( general-header ; Section 4.5
| request-header ; Section 5.3
| entity-header ) CRLF) ; Section 7.1
CRLF
[ message-body ] ; Section 4.3
5.1 Request-Line
5.1.1 Method
5.3 请求头
request-header = Accept ; Section 14.1
| Accept-Charset ; Section 14.2
| Accept-Encoding ; Section 14.3
| Accept-Language ; Section 14.4
| Authorization ; Section 14.8
| Expect ; Section 14.20
| From ; Section 14.22
| Host ; Section 14.23
| If-Match ; Section 14.24
| If-Modified-Since ; Section 14.25
| If-None-Match ; Section 14.26
| If-Range ; Section 14.27
| If-Unmodified-Since ; Section 14.28
| Max-Forwards ; Section 14.31
| Proxy-Authorization ; Section 14.34
| Range ; Section 14.35
| Referer ; Section 14.36
| TE ; Section 14.39
| User-Agent ; Section 14.43
6 Response 响应
Response = Status-Line ; Section 6.1
*(( general-header ; Section 4.5
| response-header ; Section 6.2
| entity-header ) CRLF) ; Section 7.1
CRLF
[ message-body ] ; Section 7.2
6.1
- 1xx: Informational - Request received, continuing process
- 2xx: Success - The action was successfully received,
understood, and accepted
- 3xx: Redirection - Further action must be taken in order to
complete the request
- 4xx: Client Error - The request contains bad syntax or cannot
be fulfilled
- 5xx: Server Error - The server failed to fulfill an apparently
valid request
7 Entity
7.1 Entity Header Fields
entity-header = Allow ; Section 14.7
| Content-Encoding ; Section 14.11
| Content-Language ; Section 14.12
| Content-Length ; Section 14.13
| Content-Location ; Section 14.14
| Content-MD5 ; Section 14.15
| Content-Range ; Section 14.16
| Content-Type ; Section 14.17
| Expires ; Section 14.21
| Last-Modified ; Section 14.29
| extension-header
extension-header = message-header
7.2 Entity Body
8 Connections
8.1 长链接(持久的链接)
优点
- 管理(打开、关闭)更少的链接，更少的资源(CPU/memory)占用
- 网络争抢更少
- 减少建链，降低延时
8.2 长连接是默认
9 Method Definitions 方法定义
9.2 OPTIONS
9.3 GET
9.4 HEAD
9.5 POST
9.6 PUT
9.7 DELETE
9.8 TRACE
9.9 CONNECT
10 状态码定义(Status Code Definitions)
10.1 1xx
100
101
10.2 2xx Successful
###
10.3 3xx Redirection
301 Moved Permanently
302 Found / Temporarily
303 See Other
304 Not Modified
305 Use Proxy
307 Temporary Redirect
10.4 Client Error 4xx
400 Bad Request 客户端发送数据出错
401 Unauthorized 需要带Authorization的头信息
403 Forbidden
404 Not Found
405 Method Not Allowed
406 Not Acceptable
407 Proxy Authentication Required
408 Request Timeout
409 Conflict
410 Gone
411 Length Required
412 Precondition Failed
413 Request Entity Too Large
414 Request-URI Too Long
415 Unsupported Media Type
416 Requested Range Not Satisfiable
10.5 Server Error 5xx
500 Internal Server Error
501 Not Implemented
502 Bad Gateway
503 Service Unavailable
504 Gateway Timeout
505 HTTP Version Not Supported
13 Caching in HTTP
14 Header Field Definitions
14.20 Expect
15 Security Considerations
websocket
- 2020 年 07 月 03 日
- protocal
背景
客户端&服务端双向通讯使用http问题
- a number of different underlying TCP connnections for each client
- high overhead （http header）
- client-side 需要循环获取
work over HTTP ports 80 and 443
可以看看UUID的 rfc,应该比较简单？
握手
- 服务器是否接受建立连接
- 返回的信息，client也需要检查
- 101的状态code
The |Sec-WebSocket-Accept| header field indicates whether the server is willing to accept the connection.
- 给浏览器提供一个原生的安全模型
1.7. Relationship to TCP and HTTP
The WebSocket Protocol is an independent TCP-based protocol. Its only relationship to HTTP is that its handshake is interpreted by HTTP servers as an Upgrade request.
By default, the WebSocket Protocol uses port 80 for regular WebSocket connections and port 443 for WebSocket connections tunneled over Transport Layer Security (TLS)
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-------+-+-------------+-------------------------------+
|F|R|R|R| opcode|M| Payload len | Extended payload length |
|I|S|S|S| (4) |A| (7) | (16/64) |
|N|V|V|V| |S| | (if payload len==126/127) |
| |1|2|3| |K| | |
+-+-+-+-+-------+-+-------------+ - - - - - - - - - - - - - - - +
| Extended payload length continued, if payload len == 127 |
+ - - - - - - - - - - - - - - - +-------------------------------+
| |Masking-key, if MASK set to 1 |
+-------------------------------+-------------------------------+
| Masking-key (continued) | Payload Data |
+-------------------------------- - - - - - - - - - - - - - - - +
: Payload Data continued ... :
+ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - +
| Payload Data continued ... |
+---------------------------------------------------------------+
kata container
- 2020 年 06 月 30 日
- container
keywords
- secure container
- lightweight VM
- workload isolation
- supports multiple hypervisors : QEMU, NEMU, Firecracker
优势
同时拥有
- 容器的轻量
-
虚拟机的安全隔离
- 支持kubernets
