1IPVSADM(8) Linux Administrator's Guide IPVSADM(8)
2
6 ipvsadm - Linux Virtual Server administration
7
9 ipvsadm -A|E -t|u|f service-address [-s scheduler]
10 [-p [timeout]] [-M netmask]
11 ipvsadm -D -t|u|f service-address
12 ipvsadm -C
13 ipvsadm -R
14 ipvsadm -S [-n]
15 ipvsadm -a|e -t|u|f service-address -r server-address
16 [-g|i|m] [-w weight] [-x upper] [-y lower]
17 ipvsadm -d -t|u|f service-address -r server-address
18 ipvsadm -L|l [options]
19 ipvsadm -Z [-t|u|f service-address]
20 ipvsadm --set tcp tcpfin udp
21 ipvsadm --start-daemon state [--mcast-interface interface]
22 [--syncid syncid]
23 ipvsadm --stop-daemon state
24 ipvsadm -h
25
27 Ipvsadm(8) is used to set up, maintain or inspect the virtual server
28 table in the Linux kernel. The Linux Virtual Server can be used to
29 build scalable network services based on a cluster of two or more
30 nodes. The active node of the cluster redirects service requests to a
31 collection of server hosts that will actually perform the services.
32 Supported features include two protocols (TCP and UDP), three packet-
33 forwarding methods (NAT, tunneling, and direct routing), and eight load
34 balancing algorithms (round robin, weighted round robin, least-connec‐
35 tion, weighted least-connection, locality-based least-connection,
36 locality-based least-connection with replication, destination-hashing,
37 and source-hashing).
38 The command has two basic formats for execution:
40 ipvsadm COMMAND [protocol] service-address
42 [scheduling-method] [persistence options]
43 ipvsadm command [protocol] service-address
45 server-address [packet-forwarding-method]
46 [weight options]
47 The first format manipulates a virtual service and the algorithm for
49 assigning service requests to real servers. Optionally, a persistent
50 timeout and network mask for the granularity of a persistent service
51 may be specified. The second format manipulates a real server that is
52 associated with an existing virtual service. When specifying a real
53 server, the packet-forwarding method and the weight of the real server,
54 relative to other real servers for the virtual service, may be speci‐
55 fied, otherwise defaults will be used.
56 COMMANDS
58 ipvsadm(8) recognises the commands described below. Upper-case commands
59 maintain virtual services. Lower-case commands maintain real servers
60 that are associated with a virtual service.
61 -A, --add-service
63 Add a virtual service. A service address is uniquely defined by
64 a triplet: IP address, port number, and protocol. Alternatively,
65 a virtual service may be defined by a firewall-mark.
66 -E, --edit-service
68 Edit a virtual service.
69 -D, --delete-service
71 Delete a virtual service, along with any associated real
72 servers.
73 -C, --clear
75 Clear the virtual server table.
76 -R, --restore
78 Restore Linux Virtual Server rules from stdin. Each line read
79 from stdin will be treated as the command line options to a sep‐
80 arate invocation of ipvsadm. Lines read from stdin can option‐
81 ally begin with "ipvsadm". This option is useful to avoid exe‐
82 cuting a large number or ipvsadm commands when constructing an
83 extensive routing table.
84 -S, --save
86 Dump the Linux Virtual Server rules to stdout in a format that
87 can be read by -R|--restore.
88 -a, --add-server
90 Add a real server to a virtual service.
91 -e, --edit-server
93 Edit a real server in a virtual service.
94 -d, --delete-server
96 Remove a real server from a virtual service.
97 -L, -l, --list
99 List the virtual server table if no argument is specified. If a
100 service-address is selected, list this service only. If the -c
101 option is selected, then display the connection table. The exact
102 output is affected by the other arguments given.
103 -Z, --zero
105 Zero the packet, byte and rate counters in a service or all ser‐
106 vices.
107 --set tcp tcpfin udp
109 Change the timeout values used for IPVS connections. This com‐
110 mand always takes 3 parameters, representing the timeout
111 values (in seconds) for TCP sessions, TCP sessions after receiv‐
112 ing a FIN packet, and UDP packets, respectively. A timeout
113 value 0 means that the current timeout value of the correspond‐
114 ing entry is preserved.
115 --start-daemon state
117 Start the connection synchronization daemon. The state is to
118 indicate that the daemon is started as master or backup. The
119 connection synchronization daemon is implemented inside the
120 Linux kernel. The master daemon running at the primary load bal‐
121 ancer multicasts changes of connections periodically, and the
122 backup daemon running at the backup load balancers receives mul‐
123 ticast message and creates corresponding connections. Then, in
124 case the primary load balancer fails, a backup load balancer
125 will takeover, and it has state of almost all connections, so
126 that almost all established connections can continue to access
127 the service.
128 --stop-daemon
130 Stop the connection synchronization daemon.
131 -h, --help
133 Display a description of the command syntax.
134 PARAMETERS
136 The commands above accept or require zero or more of the following
137 parameters.
138 -t, --tcp-service service-address
140 Use TCP service. The service-address is of the form host[:port].
141 Host may be one of a plain IP address or a hostname. Port may be
142 either a plain port number or the service name of port. The Port
143 may be omitted, in which case zero will be used. A Port of zero
144 is only valid if the service is persistent as the -p|--persis‐
145 tent option, in which case it is a wild-card port, that is con‐
146 nections will be accepted to any port.
147 -u, --udp-service service-address
149 Use UDP service. See the -t|--tcp-service for the description of
150 the service-address.
151 -f, --fwmark-service integer
153 Use a firewall-mark, an integer value greater than zero, to
154 denote a virtual service instead of an address, port and proto‐
155 col (UDP or TCP). The marking of packets with a firewall-mark is
156 configured using the -m|--mark option to iptables(8). It can be
157 used to build a virtual service assoicated with the same real
158 servers, covering multiple IP address, port and protocol trip‐
159 plets.
160 Using firewall-mark virtual services provides a convenient
162 method of grouping together different IP addresses, ports and
163 protocols into a single virtual service. This is useful for both
164 simplifying configuration if a large number of virtual services
165 are required and grouping persistence across what would other‐
166 wise be multiple virtual services.
167 -s, --scheduler scheduling-method
169 scheduling-method Algorithm for allocating TCP connections and
170 UDP datagrams to real servers. Scheduling algorithms are imple‐
171 mented as kernel modules. Ten are shipped with the Linux Virtual
172 Server:
173 rr - Robin Robin: distributes jobs equally amongst the available
175 real servers.
176 wrr - Weighted Round Robin: assigns jobs to real servers propor‐
178 tionally to there real servers' weight. Servers with higher
179 weights receive new jobs first and get more jobs than servers
180 with lower weights. Servers with equal weights get an equal dis‐
181 tribution of new jobs.
182 lc - Least-Connection: assigns more jobs to real servers with
184 fewer active jobs.
185 wlc - Weighted Least-Connection: assigns more jobs to servers
187 with fewer jobs and relative to the real servers' weight
188 (Ci/Wi). This is the default.
189 lblc - Locality-Based Least-Connection: assigns jobs destined
191 for the same IP address to the same server if the server is not
192 overloaded and available; otherwise assign jobs to servers with
193 fewer jobs, and keep it for future assignment.
194 lblcr - Locality-Based Least-Connection with Replication:
196 assigns jobs destined for the same IP address to the least-con‐
197 nection node in the server set for the IP address. If all the
198 node in the server set are over loaded, it picks up a node with
199 fewer jobs in the cluster and adds it in the sever set for the
200 target. If the server set has not been modified for the speci‐
201 fied time, the most loaded node is removed from the server set,
202 in order to avoid high degree of replication.
203 dh - Destination Hashing: assigns jobs to servers through look‐
205 ing up a statically assigned hash table by their destination IP
206 addresses.
207 sh - Source Hashing: assigns jobs to servers through looking up
209 a statically assigned hash table by their source IP addresses.
210 sed - Shortest Expected Delay: assigns an incoming job to the
212 server with the shortest expected delay. The expected delay that
213 the job will experience is (Ci + 1) / Ui if sent to the ith
214 server, in which Ci is the number of jobs on the the ith server
215 and Ui is the fixed service rate (weight) of the ith server.
216 nq - Never Queue: assigns an incoming job to an idle server if
218 there is, instead of waiting for a fast one; if all the servers
219 are busy, it adopts the Shortest Expected Delay policy to assign
220 the job.
221 -p, --persistent [timeout]
223 Specify that a virtual service is persistent. If this option is
224 specified, multiple requests from a client are redirected to the
225 same real server selected for the first request. Optionally,
226 the timeout of persistent sessions may be specified given in
227 seconds, otherwise the default of 300 seconds will be used. This
228 option may be used in conjunction with protocols such as SSL or
229 FTP where it is important that clients consistently connect with
230 the same real server.
231 Note: If a virtual service is to handle FTP connections then
233 persistence must be set for the virtual service if Direct Rout‐
234 ing or Tunnelling is used as the forwarding mechanism. If Mas‐
235 querading is used in conjunction with an FTP service than per‐
236 sistence is not necessary, but the ip_vs_ftp kernel module must
237 be used. This module may be manually inserted into the kernel
238 using insmod(8).
239 -M, --netmask netmask
241 Specify the granularity with which clients are grouped for per‐
242 sistent virtual services. The source address of the request is
243 masked with this netmask to direct all clients from a network to
244 the same real server. The default is 255.255.255.255, that is,
245 the persistence granularity is per client host. Less specific
246 netmasks may be used to resolve problems with non-persistent
247 cache clusters on the client side.
248 -r, --real-server server-address
250 Real server that an associated request for service may be
251 assigned to. The server-address is the host address of a real
252 server, and may plus port. Host can be either a plain IP address
253 or a hostname. Port can be either a plain port number or the
254 service name of port. In the case of the masquerading method,
255 the host address is usually an RFC 1918 private IP address, and
256 the port can be different from that of the associated service.
257 With the tunneling and direct routing methods, port must be
258 equal to that of the service address. For normal services, the
259 port specified in the service address will be used if port is
260 not specified. For fwmark services, port may be omitted, in
261 which case the destination port on the real server will be the
262 destination port of the request sent to the virtual service.
263 [packet-forwarding-method]
265 -g, --gatewaying Use gatewaying (direct routing). This is the
267 default.
268 -i, --ipip Use ipip encapsulation (tunneling).
270 -m, --masquerading Use masquerading (network access transla‐
272 tion, or NAT).
273 Note: Regardless of the packet-forwarding mechanism specified,
275 real servers for addresses for which there are interfaces on the
276 local node will be use the local forwarding method, then packets
277 for the servers will be passed to upper layer on the local node.
278 This cannot be specified by ipvsadm, rather it set by the kernel
279 as real servers are added or modified.
280 -w, --weight weight
282 Weight is an integer specifying the capacity of a server rela‐
283 tive to the others in the pool. The valid values of weight are 0
284 through to 65535. The default is 1. Quiescent servers are speci‐
285 fied with a weight of zero. A quiescent server will receive no
286 new jobs but still serve the existing jobs, for all scheduling
287 algorithms distributed with the Linux Virtual Server. Setting a
288 quiescent server may be useful if the server is overloaded or
289 needs to be taken out of service for maintenance.
290 -x, --u-threshold uthreshold
292 uthreshold is an integer specifying the upper connection thresh‐
293 old of a server. The valid values of uthreshold are 0 through to
294 65535. The default is 0, which means the upper connection
295 threshold is not set. If uthreshold is set with other values, no
296 new connections will be sent to the server when the number of
297 its connections exceeds its upper connection threshold.
298 -y, --l-threshold lthreshold
300 lthreshold is an integer specifying the lower connection thresh‐
301 old of a server. The valid values of lthreshold are 0 through to
302 65535. The default is 0, which means the lower connection
303 threshold is not set. If lthreshold is set with other values,
304 the server will receive new connections when the number of its
305 connections drops below its lower connection threshold. If
306 lthreshold is not set but uthreshold is set, the server will
307 receive new connections when the number of its connections drops
308 below three forth of its upper connection threshold.
309 --mcast-interface interface
311 Specify the multicast interface that the sync master daemon
312 sends outgoing multicasts through, or the sync backup daemon
313 listens to for multicasts.
314 --syncid syncid
316 Specify the syncid that the sync master daemon fills in the Syn‐
317 cID header while sending multicast messages, or the sync backup
318 daemon uses to filter out multicast messages not matched with
319 the SyncID value. The valid values of syncid are 0 through to
320 255. The default is 0, which means no filtering at all.
321 -c, --connection
323 Connection output. The list command with this option will list
324 current IPVS connections.
325 --timeout
327 Timeout output. The list command with this option will display
328 the timeout values (in seconds) for TCP sessions, TCP sessions
329 after receiving a FIN packet, and UDP packets.
330 --daemon
332 Daemon information output. The list command with this option
333 will display the daemon status and its multicast interface.
334 --stats
336 Output of statistics information. The list command with this
337 option will display the statistics information of services and
338 their servers.
339 --rate Output of rate information. The list command with this option
341 will display the rate information (such as connections/second,
342 bytes/second and packets/second) of services and their servers.
343 --thresholds
345 Output of thresholds information. The list command with this
346 option will display the upper/lower connection threshold infor‐
347 mation of each server in service listing.
348 --persistent-conn
350 Output of persistent connection information. The list command
351 with this option will display the persistent connection counter
352 information of each server in service listing. The persistent
353 connection is used to forward the actual connections from the
354 same client/network to the same server.
355 --sort Sort the list of virtual services and real servers. The virtual
357 service entries are sorted in ascending order by <protocol,
358 address, port>. The real server entries are sorted in ascending
359 order by <address, port>.
360 -n, --numeric
362 Numeric output. IP addresses and port numbers will be printed
363 in numeric format rather than as as host names and services
364 respectively, which is the default.
365
367 The following commands configure a Linux Director to distribute incom‐
368 ing requests addressed to port 80 on 207.175.44.110 equally to port 80
369 on five real servers. The forwarding method used in this example is
370 NAT, with each of the real servers being masqueraded by the Linux
371 Director.
372 ipvsadm -A -t 207.175.44.110:80 -s rr
374 ipvsadm -a -t 207.175.44.110:80 -r 192.168.10.1:80 -m
375 ipvsadm -a -t 207.175.44.110:80 -r 192.168.10.2:80 -m
376 ipvsadm -a -t 207.175.44.110:80 -r 192.168.10.3:80 -m
377 ipvsadm -a -t 207.175.44.110:80 -r 192.168.10.4:80 -m
378 ipvsadm -a -t 207.175.44.110:80 -r 192.168.10.5:80 -m
379 Alternatively, this could be achieved in a single ipvsadm command.
381 echo "
383 -A -t 207.175.44.110:80 -s rr
384 -a -t 207.175.44.110:80 -r 192.168.10.1:80 -m
385 -a -t 207.175.44.110:80 -r 192.168.10.2:80 -m
386 -a -t 207.175.44.110:80 -r 192.168.10.3:80 -m
387 -a -t 207.175.44.110:80 -r 192.168.10.4:80 -m
388 -a -t 207.175.44.110:80 -r 192.168.10.5:80 -m
389 " | ipvsadm -R
390 As masquerading is used as the forwarding mechanism in this example,
392 the default route of the real servers must be set to the linux direc‐
393 tor, which will need to be configured to forward and masquerade pack‐
394 ets. This can be achieved using the following commands:
395 echo "1" > /proc/sys/net/ipv4/ip_forward
397
399 The following commands configure a Linux Director to distribute incom‐
400 ing requests addressed to any port on 207.175.44.110 or 207.175.44.111
401 equally to the corresponding port on five real servers. As per the pre‐
402 vious example, the forwarding method used in this example is NAT, with
403 each of the real servers being masqueraded by the Linux Director.
404 ipvsadm -A -f 1 -s rr
406 ipvsadm -a -f 1 -r 192.168.10.1:0 -m
407 ipvsadm -a -f 1 -r 192.168.10.2:0 -m
408 ipvsadm -a -f 1 -r 192.168.10.3:0 -m
409 ipvsadm -a -f 1 -r 192.168.10.4:0 -m
410 ipvsadm -a -f 1 -r 192.168.10.5:0 -m
411 As masquerading is used as the forwarding mechanism in this example,
413 the default route of the real servers must be set to the linux direc‐
414 tor, which will need to be configured to forward and masquerade pack‐
415 ets. The real server should also be configured to mark incoming packets
416 addressed to any port on 207.175.44.110 and 207.175.44.111 with fire‐
417 wall-mark 1. If FTP traffic is to be handled by this virtual service,
418 then the ip_vs_ftp kernel module needs to be inserted into the kernel.
419 These operations can be achieved using the following commands:
420 echo "1" > /proc/sys/net/ipv4/ip_forward
422 modprobe ip_tables
423 iptables -A PREROUTING -t mangle -d 207.175.44.110/31 -j MARK --set-mark 1
424 modprobe ip_vs_ftp
425
427 The Linux Virtual Server implements three defense strategies against
428 some types of denial of service (DoS) attacks. The Linux Director cre‐
429 ates an entry for each connection in order to keep its state, and each
430 entry occupies 128 bytes effective memory. LVS's vulnerability to a DoS
431 attack lies in the potential to increase the number entries as much as
432 possible until the linux director runs out of memory. The three defense
433 strategies against the attack are: Randomly drop some entries in the
434 table. Drop 1/rate packets before forwarding them. And use secure tcp
435 state transition table and short timeouts. The strategies are con‐
436 trolled by sysctl variables and corresponding entries in the /proc
437 filesystem:
438 /proc/sys/net/ipv4/vs/drop_entry /proc/sys/net/ipv4/vs/drop_packet
440 /proc/sys/net/ipv4/vs/secure_tcp
441 Valid values for each variable are 0 through to 3. The default value is
443 0, which disables the respective defense strategy. 1 and 2 are auto‐
444 matic modes - when there is no enough available memory, the respective
445 strategy will be enabled and the variable is automatically set to 2,
446 otherwise the strategy is disabled and the variable is set to 1. A
447 value of 3 denotes that the respective strategy is always enabled. The
448 available memory threshold and secure TCP timeouts can be tuned using
449 the sysctl variables and corresponding entries in the /proc filesystem:
450 /proc/sys/net/ipv4/vs/amemthresh /proc/sys/net/ipv4/vs/timeout_*
452
454 /proc/net/ip_vs
455 /proc/net/ip_vs_app
456 /proc/net/ip_vs_conn
457 /proc/net/ip_vs_stats
458 /proc/sys/net/ipv4/vs/am_droprate
459 /proc/sys/net/ipv4/vs/amemthresh
460 /proc/sys/net/ipv4/vs/drop_entry
461 /proc/sys/net/ipv4/vs/drop_packet
462 /proc/sys/net/ipv4/vs/secure_tcp
463 /proc/sys/net/ipv4/vs/timeout_close
464 /proc/sys/net/ipv4/vs/timeout_closewait
465 /proc/sys/net/ipv4/vs/timeout_established
466 /proc/sys/net/ipv4/vs/timeout_finwait
467 /proc/sys/net/ipv4/vs/timeout_icmp
468 /proc/sys/net/ipv4/vs/timeout_lastack
469 /proc/sys/net/ipv4/vs/timeout_listen
470 /proc/sys/net/ipv4/vs/timeout_synack
471 /proc/sys/net/ipv4/vs/timeout_synrecv
472 /proc/sys/net/ipv4/vs/timeout_synsent
473 /proc/sys/net/ipv4/vs/timeout_timewait
474 /proc/sys/net/ipv4/vs/timeout_udp
475
477 The LVS web site (http://www.linuxvirtualserver.org/) for more documen‐
478 tation about LVS.
479 ipvsadm-save(8), ipvsadm-restore(8), iptables(8),
481 insmod(8), modprobe(8)
482
484 ipvsadm - Wensong Zhang <wensong@linuxvirtualserver.org>
485 Peter Kese <peter.kese@ijs.si>
486 man page - Mike Wangsmo <wanger@redhat.com>
487 Wensong Zhang <wensong@linuxvirtualserver.org>
488 Horms <horms@verge.net.au>
4894th Berkeley Distribution 5th July 2003 IPVSADM(8)