TiDB & Haproxy / show processlist View real IP

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

global
    log /dev/log local0
    maxconn 65536
    ulimit-n 131086
    quiet
    pidfile /var/run/haproxy.pid

defaults
    option httplog
    option redispatch
    option nolinger
    option dontlognull
    retries 3
    contimeout 5000
    clitimeout 50000
    srvtimeout 50000
    log 127.0.0.1 local3

listen testip
    bind :3308
    mode tcp
    balance roundrobin
    server tidb2 10.0.1.4:4001 weight 1

1
2
3
4
5
6
7

mysql> show processlist;
+------+------+--------------+------+---------+------+-------+------------------+
| Id   | User | Host         | db   | Command | Time | State | Info             |
+------+------+--------------+------+---------+------+-------+------------------+
|    1 | st   | 42.X.X.X |      | Query   |    0 | 2     | show processlist |
+------+------+--------------+------+---------+------+-------+------------------+
1 row in set (0.00 sec)

1
2
3
4
5
6
7
8

mysql> show processlist;
+------+------+--------------+------+---------+------+-------+------------------+
| Id   | User | Host         | db   | Command | Time | State | Info             |
+------+------+--------------+------+---------+------+-------+------------------+
|    1 | st   | 42.X.X.X |      | Sleep   |  597 | 2     |                  |
|    2 | st   | 10.0.1.4     |      | Query   |    0 | 2     | show processlist |
+------+------+--------------+------+---------+------+-------+------------------+
2 rows in set (0.00 sec)

1
2
3
4
5
6
7

[proxy-protocol]
# PROXY protocol acceptable client networks.
# Empty string means disable PROXY protocol, * means all networks.
networks = ""

# PROXY protocol header read timeout, unit is second
header-timeout = 5

1
2
3
4
5

Add PROXY protocol V1 and V2 support.
usage: tidb-server --proxy-protocol-networks "*" --proxy-protocol-header-timeout 5
Add --proxy-protocol-networks command parameter for PROXY protocol enable or disable. If you want to limit HAProxy server IP range, you can set --proxy-protocol-networks parameter to a CIDRs and split by ",". For example:
tidb-server --proxy-protocol-networks "192.168.1.0/24,192.168.2.0/24"
For more information about PROXY protocol please refer https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

global
    log /dev/log local0
    maxconn 65536
    ulimit-n 131086
    quiet
    pidfile /var/run/haproxy.pid

defaults
    option httplog
    option redispatch
    option nolinger
    option dontlognull
    retries 3
    contimeout 5000
    clitimeout 50000
    srvtimeout 50000
    log 10.0.1.4 local3

listen web
    bind :3307
    mode tcp
    balance roundrobin
    server tidb1 10.0.1.4:4000 send-proxy weight 1

1
2
3
4
5
6
7
8

mysql> show processlist;
+------+------+--------------+------+---------+------+-------+------------------+
| Id   | User | Host         | db   | Command | Time | State | Info             |
+------+------+--------------+------+---------+------+-------+------------------+
|    3 | te   | 42.X.X.X |      | Query   |    0 | 2     | show processlist |
|    4 | te   | 42.X.X.X |      | Query   |    6 | 2     | select sleep(100) |
+------+------+--------------+------+---------+------+-------+------------------+
1 row in set (0.00 sec)

1
2
3
4
5
6
7
8

mysql> show processlist;
+------+------+--------------+------+---------+------+-------+-------------------+
| Id   | User | Host         | db   | Command | Time | State | Info              |
+------+------+--------------+------+---------+------+-------+-------------------+
|    4 | te   | 42.X.X.X |      | Query   |    6 | 15    | select sleep(100) |
|    5 | te   | 42.X.X.X |      | Query   |    0 | 2     | show processlist  |
+------+------+--------------+------+---------+------+-------+-------------------+
2 rows in set (0.00 sec)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

1. most proxy protocol implementations such as MariaDB, Percona server, apache(via mod_remoteip), allow user to specify a list of cider blocks（`proxy-protocol-networks` for MariaDB and Percona server，`RemoteIPProxyProtocolExceptions ` for apache), IP in the list MUST use proxy protocol and IP not in the list MUST NOT use proxy protocol(but they can still connect without proxy protocol). some implementations, like nginx and TiDB, forces all IP addresses to use proxy protocol when enables proxy protocol options.
2. the spec says that we "MUST not try to guess whether the protocol header is presentor not", to prevent security issues, but there's no guessing when we provide a list and config IPs in the list to MUST use and IPs not in the list to MUST NOT use(but still able to connect). There's no security issues doing this way and it actually meet the spec requirement.
3. allowing certain IP blocks to connect directly without haproxy will make certain tasks much easier to configure, such as:
    * monitoring individual TiDB nodes
    * make a certain TiDB node to do complex OLAP tasks without influence other OLTP nodes.
    * control individual TiDB nodes via script tasks in the local machine(such as a crontab script at each TiDB node

I think make IPs not in AllowedIPs to connect directly without proxy protocol is:

1. very easy to implement. only need to modify two lines of code
2. has some benefits
3. meets the spec

So I think its good to do this way.

1
2
3
4
5

listen web
    bind :3307
    mode tcp
    balance roundrobin
    server tidb1 127.0.0.1:4000 send-proxy weight 1

1
2
3
4
5

listen web
    bind :3307
    mode tcp
    balance roundrobin
    server tidb1 10.0.1.4:4000 send-proxy weight 1

1
2
3
4
5

listen web
    bind :3307
    mode tcp
    balance roundrobin
    server tidb1 10.0.1.4:4000 send-proxy weight 1

1
2
3
4
5

listen web
    bind :3308
    mode tcp
    balance roundrobin
    server tidb1 10.0.1.4:4001 weight 1

1

[2020/01/09 22:02:59.274 +08:00] [ERROR] [server.go:321] ["PROXY protocol failed"] [error="Header read timeout"] [stack="github.com/pingcap/tidb/server.(*Server).Run\n\t/home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/server/server.go:321\nmain.runServer\n\t/home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/tidb-server/main.go:568\nmain.main\n\t/home/jenkins/workspace/build_tidb_master/go/src/github.com/pingcap/tidb/tidb-server/main.go:174\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:200"]