1IPVSADM(8) Linux Administrator's Guide IPVSADM(8)
2
6 ipvsadm - Linux Virtual Server administration
7
9 ipvsadm -A|E virtual-service [-s scheduler]
10 [-p [timeout]] [-M netmask] [-b sched-flags]
11 ipvsadm -D virtual-service
12 ipvsadm -C
13 ipvsadm -R
14 ipvsadm -S [-n]
15 ipvsadm -a|e virtual-service -r server-address
16 [-g|i|m] [-w weight] [-x upper] [-y lower]
17 ipvsadm -d virtual-service -r server-address
18 ipvsadm -L|l [virtual-service] [options]
19 ipvsadm -Z [virtual-service]
20 ipvsadm --set tcp tcpfin udp
21 ipvsadm --start-daemon state [daemon-options]
22 [--syncid syncid]
23 ipvsadm --stop-daemon state
24 ipvsadm -h
25
27 Ipvsadm(8) is used to set up, maintain or inspect the virtual server
28 table in the Linux kernel. The Linux Virtual Server can be used to
29 build scalable network services based on a cluster of two or more
30 nodes. The active node of the cluster redirects service requests to a
31 collection of server hosts that will actually perform the services.
32 Supported features include three protocols (TCP, UDP and SCTP), three
33 packet-forwarding methods (NAT, tunneling, and direct routing), and
34 eight load balancing algorithms (round robin, weighted round robin,
35 least-connection, weighted least-connection, locality-based least-con‐
36 nection, locality-based least-connection with replication, destination-
37 hashing, and source-hashing).
38 The command has two basic formats for execution:
40 ipvsadm COMMAND virtual-service
42 [scheduling-method] [persistence options]
43 ipvsadm command virtual-service
45 server-address [packet-forwarding-method]
46 [weight options]
47 The first format manipulates a virtual service and the algorithm for
49 assigning service requests to real servers. Optionally, a persistent
50 timeout and network mask for the granularity of a persistent service
51 may be specified. The second format manipulates a real server that is
52 associated with an existing virtual service. When specifying a real
53 server, the packet-forwarding method and the weight of the real server,
54 relative to other real servers for the virtual service, may be speci‐
55 fied, otherwise defaults will be used.
56 COMMANDS
58 ipvsadm(8) recognises the commands described below. Upper-case commands
59 maintain virtual services. Lower-case commands maintain real servers
60 that are associated with a virtual service.
61 -A, --add-service
63 Add a virtual service. A service address is uniquely defined by
64 a triplet: IP address, port number, and protocol. Alternatively,
65 a virtual service may be defined by a firewall-mark.
66 -E, --edit-service
68 Edit a virtual service.
69 -D, --delete-service
71 Delete a virtual service, along with any associated real
72 servers.
73 -C, --clear
75 Clear the virtual server table.
76 -R, --restore
78 Restore Linux Virtual Server rules from stdin. Each line read
79 from stdin will be treated as the command line options to a sep‐
80 arate invocation of ipvsadm. Lines read from stdin can option‐
81 ally begin with "ipvsadm". This option is useful to avoid exe‐
82 cuting a large number or ipvsadm commands when constructing an
83 extensive routing table.
84 -S, --save
86 Dump the Linux Virtual Server rules to stdout in a format that
87 can be read by -R|--restore.
88 -a, --add-server
90 Add a real server to a virtual service.
91 -e, --edit-server
93 Edit a real server in a virtual service.
94 -d, --delete-server
96 Remove a real server from a virtual service.
97 -L, -l, --list
99 List the virtual server table if no argument is specified. If a
100 service-address is selected, list this service only. If the -c
101 option is selected, then display the connection table. The exact
102 output is affected by the other arguments given.
103 -Z, --zero
105 Zero the packet, byte and rate counters in a service or all ser‐
106 vices.
107 --set tcp tcpfin udp
109 Change the timeout values used for IPVS connections. This com‐
110 mand always takes 3 parameters, representing the timeout
111 values (in seconds) for TCP sessions, TCP sessions after receiv‐
112 ing a FIN packet, and UDP packets, respectively. A timeout
113 value 0 means that the current timeout value of the correspond‐
114 ing entry is preserved.
115 --start-daemon state
117 Start the connection synchronization daemon. The state is to
118 indicate that the daemon is started as master or backup. The
119 connection synchronization daemon is implemented inside the
120 Linux kernel. The master daemon running at the primary load bal‐
121 ancer multicasts changes of connections periodically, and the
122 backup daemon running at the backup load balancers receives mul‐
123 ticast message and creates corresponding connections. Then, in
124 case the primary load balancer fails, a backup load balancer
125 will takeover, and it has state of almost all connections, so
126 that almost all established connections can continue to access
127 the service.
128 The sync daemon supports IPv4 and IPv6 connections.
130 --stop-daemon
132 Stop the connection synchronization daemon.
133 -h, --help
135 Display a description of the command syntax.
136 virtual-service
138 Specifies the virtual service based on protocol/addr/port or firewall
139 mark.
140 -t, --tcp-service service-address
142 Use TCP service. The service-address is of the form host[:port].
143 Host may be one of a plain IP address or a hostname. Port may be
144 either a plain port number or the service name of port. The Port
145 may be omitted, in which case zero will be used. A Port of zero
146 is only valid if the service is persistent as the -p|--persis‐
147 tent option, in which case it is a wild-card port, that is con‐
148 nections will be accepted to any port.
149 -u, --udp-service service-address
151 Use UDP service. See the -t|--tcp-service for the description of
152 the service-address.
153 --sctp-service service-address
155 Use SCTP service. See the -t|--tcp-service for the description
156 of the service-address.
157 -f, --fwmark-service integer
159 Use a firewall-mark, an integer value greater than zero, to
160 denote a virtual service instead of an address, port and proto‐
161 col (UDP or TCP). The marking of packets with a firewall-mark is
162 configured using the -m|--mark option to iptables(8). It can be
163 used to build a virtual service associated with the same real
164 servers, covering multiple IP address, port and protocol
165 triplets. If IPv6 addresses are used, the -6 option must be
166 used.
167 Using firewall-mark virtual services provides a convenient
169 method of grouping together different IP addresses, ports and
170 protocols into a single virtual service. This is useful for both
171 simplifying configuration if a large number of virtual services
172 are required and grouping persistence across what would other‐
173 wise be multiple virtual services.
174 PARAMETERS
176 The commands above accept or require zero or more of the following
177 parameters.
178 -s, --scheduler scheduling-method
180 scheduling-method Algorithm for allocating TCP connections and
181 UDP datagrams to real servers. Scheduling algorithms are imple‐
182 mented as kernel modules. Ten are shipped with the Linux Virtual
183 Server:
184 rr - Round Robin: distributes jobs equally amongst the available
186 real servers.
187 wrr - Weighted Round Robin: assigns jobs to real servers propor‐
189 tionally to there real servers' weight. Servers with higher
190 weights receive new jobs first and get more jobs than servers
191 with lower weights. Servers with equal weights get an equal dis‐
192 tribution of new jobs.
193 lc - Least-Connection: assigns more jobs to real servers with
195 fewer active jobs.
196 wlc - Weighted Least-Connection: assigns more jobs to servers
198 with fewer jobs and relative to the real servers' weight
199 (Ci/Wi). This is the default.
200 lblc - Locality-Based Least-Connection: assigns jobs destined
202 for the same IP address to the same server if the server is not
203 overloaded and available; otherwise assign jobs to servers with
204 fewer jobs, and keep it for future assignment.
205 lblcr - Locality-Based Least-Connection with Replication:
207 assigns jobs destined for the same IP address to the least-con‐
208 nection node in the server set for the IP address. If all the
209 node in the server set are over loaded, it picks up a node with
210 fewer jobs in the cluster and adds it in the sever set for the
211 target. If the server set has not been modified for the speci‐
212 fied time, the most loaded node is removed from the server set,
213 in order to avoid high degree of replication.
214 dh - Destination Hashing: assigns jobs to servers through look‐
216 ing up a statically assigned hash table by their destination IP
217 addresses.
218 sh - Source Hashing: assigns jobs to servers through looking up
220 a statically assigned hash table by their source IP addresses.
221 This scheduler has two flags: sh-fallback, which enables fall‐
222 back to a different server if the selected server was unavail‐
223 able, and sh-port, which adds the source port number to the hash
224 computation.
225 sed - Shortest Expected Delay: assigns an incoming job to the
227 server with the shortest expected delay. The expected delay that
228 the job will experience is (Ci + 1) / Ui if sent to the ith
229 server, in which Ci is the number of jobs on the the ith server
230 and Ui is the fixed service rate (weight) of the ith server.
231 nq - Never Queue: assigns an incoming job to an idle server if
233 there is, instead of waiting for a fast one; if all the servers
234 are busy, it adopts the Shortest Expected Delay policy to assign
235 the job.
236 -p, --persistent [timeout]
238 Specify that a virtual service is persistent. If this option is
239 specified, multiple requests from a client are redirected to the
240 same real server selected for the first request. Optionally,
241 the timeout of persistent sessions may be specified given in
242 seconds, otherwise the default of 300 seconds will be used. This
243 option may be used in conjunction with protocols such as SSL or
244 FTP where it is important that clients consistently connect with
245 the same real server.
246 Note: If a virtual service is to handle FTP connections then
248 persistence must be set for the virtual service if Direct Rout‐
249 ing or Tunnelling is used as the forwarding mechanism. If Mas‐
250 querading is used in conjunction with an FTP service than per‐
251 sistence is not necessary, but the ip_vs_ftp kernel module must
252 be used. This module may be manually inserted into the kernel
253 using insmod(8).
254 -M, --netmask netmask
256 Specify the granularity with which clients are grouped for per‐
257 sistent virtual services. The source address of the request is
258 masked with this netmask to direct all clients from a network to
259 the same real server. The default is 255.255.255.255, that is,
260 the persistence granularity is per client host. Less specific
261 netmasks may be used to resolve problems with non-persistent
262 cache clusters on the client side. IPv6 netmasks should be
263 specified as a prefix length between 1 and 128. The default
264 prefix length is 128.
265 -b, --sched-flags sched-flags
267 Set scheduler flags for this virtual server. sched-flags is a
268 comma-separated list of flags. See the scheduler descriptions
269 for valid scheduler flags.
270 -r, --real-server server-address
272 Real server that an associated request for service may be
273 assigned to. The server-address is the host address of a real
274 server, and may plus port. Host can be either a plain IP address
275 or a hostname. Port can be either a plain port number or the
276 service name of port. In the case of the masquerading method,
277 the host address is usually an RFC 1918 private IP address, and
278 the port can be different from that of the associated service.
279 With the tunneling and direct routing methods, port must be
280 equal to that of the service address. For normal services, the
281 port specified in the service address will be used if port is
282 not specified. For fwmark services, port may be omitted, in
283 which case the destination port on the real server will be the
284 destination port of the request sent to the virtual service.
285 [packet-forwarding-method]
287 -g, --gatewaying Use gatewaying (direct routing). This is the
289 default.
290 -i, --ipip Use ipip encapsulation (tunneling).
292 -m, --masquerading Use masquerading (network access transla‐
294 tion, or NAT).
295 Note: Regardless of the packet-forwarding mechanism specified,
297 real servers for addresses for which there are interfaces on the
298 local node will be use the local forwarding method, then packets
299 for the servers will be passed to upper layer on the local node.
300 This cannot be specified by ipvsadm, rather it set by the kernel
301 as real servers are added or modified.
302 -w, --weight weight
304 Weight is an integer specifying the capacity of a server rela‐
305 tive to the others in the pool. The valid values of weight are 0
306 through to 65535. The default is 1. Quiescent servers are speci‐
307 fied with a weight of zero. A quiescent server will receive no
308 new jobs but still serve the existing jobs, for all scheduling
309 algorithms distributed with the Linux Virtual Server. Setting a
310 quiescent server may be useful if the server is overloaded or
311 needs to be taken out of service for maintenance.
312 -x, --u-threshold uthreshold
314 uthreshold is an integer specifying the upper connection thresh‐
315 old of a server. The valid values of uthreshold are 0 through to
316 65535. The default is 0, which means the upper connection
317 threshold is not set. If uthreshold is set with other values, no
318 new connections will be sent to the server when the number of
319 its connections exceeds its upper connection threshold.
320 -y, --l-threshold lthreshold
322 lthreshold is an integer specifying the lower connection thresh‐
323 old of a server. The valid values of lthreshold are 0 through to
324 65535. The default is 0, which means the lower connection
325 threshold is not set. If lthreshold is set with other values,
326 the server will receive new connections when the number of its
327 connections drops below its lower connection threshold. If
328 lthreshold is not set but uthreshold is set, the server will
329 receive new connections when the number of its connections drops
330 below three forth of its upper connection threshold.
331 -c, --connection
333 Connection output. The list command with this option will list
334 current IPVS connections.
335 --timeout
337 Timeout output. The list command with this option will display
338 the timeout values (in seconds) for TCP sessions, TCP sessions
339 after receiving a FIN packet, and UDP packets.
340 --daemon
342 Daemon information output. The list command with this option
343 will display the daemon status and its multicast interface.
344 --stats
346 Output of statistics information. The list command with this
347 option will display the statistics information of services and
348 their servers.
349 --rate Output of rate information. The list command with this option
351 will display the rate information (such as connections/second,
352 bytes/second and packets/second) of services and their servers.
353 --thresholds
355 Output of thresholds information. The list command with this
356 option will display the upper/lower connection threshold infor‐
357 mation of each server in service listing.
358 --persistent-conn
360 Output of persistent connection information. The list command
361 with this option will display the persistent connection counter
362 information of each server in service listing. The persistent
363 connection is used to forward the actual connections from the
364 same client/network to the same server.
365 The list command with the -c, --connection option and this
367 option will include persistence engine data, if any is present,
368 when listing connections.
369 --sort Sort the list of virtual services and real servers. The virtual
371 service entries are sorted in ascending order by <protocol,
372 address, port>. The real server entries are sorted in ascending
373 order by <address, port>. (default)
374 --nosort
376 Do not sort the list of virtual services and real servers.
377 -n, --numeric
379 Numeric output. IP addresses and port numbers will be printed
380 in numeric format rather than as as host names and services
381 respectively, which is the default.
382 --exact
384 Expand numbers. Display the exact value of the packet and byte
385 counters, instead of only the rounded number in K's (multiples
386 of 1000) M's (multiples of 1000K) or G's (multiples of 1000M).
387 This option is only relevant for the -L command.
388 -6, --ipv6
390 Use with -f to signify fwmark rule uses IPv6 addresses.
391 -o, --ops
393 One-packet scheduling. Used in conjunction with a UDP virtual
394 service or a fwmark virtual service that handles only UDP pack‐
395 ets. All connections are created such that they only schedule
396 one packet.
397 PARAMETERS FOR SYNCHRONIZATION DAEMON
399 The --start-daemon requires zero or more of the following parameters.
400 --syncid syncid
402 Specify the syncid that the sync master daemon fills in the Syn‐
403 cID header while sending multicast messages, or the sync backup
404 daemon uses to filter out multicast messages not matched with
405 the SyncID value. The valid values of syncid are 0 through to
406 255. The default is 0, which means no filtering at all.
407 --sync-maxlen length
409 Specify the desired length of sync messages (UDP payload size).
410 It is expected that backup server will use value not less than
411 the used value in master server. The valid values of length are
412 in the 1 .. (65535 - 20 - 8) range but the kernel ensures a
413 space for at least one sync message. If value is lower than MTU
414 the sync messages will be fragmented by IP layer. The default
415 value is derived from the MTU value when daemon is started but
416 master daemon will not default to value above 1500 for compati‐
417 bility reasons.
418 --mcast-interface interface
420 Specify the multicast interface that the sync master daemon
421 sends outgoing multicasts through, or the sync backup daemon
422 listens to for multicasts.
423 --mcast-group address
425 Specify IPv4 or IPv6 multicast address for the sync messages.
426 The default value is 224.0.0.81.
427 --mcast-port port
429 Specify the UDP port for sync messages. The default value is
430 8848.
431 --mcast-ttl ttl
433 Specify the TTL value for sync messages (1 .. 255). The default
434 value is 1.
435
437 The following commands configure a Linux Director to distribute incom‐
438 ing requests addressed to port 80 on 207.175.44.110 equally to port 80
439 on five real servers. The forwarding method used in this example is
440 NAT, with each of the real servers being masqueraded by the Linux
441 Director.
442 ipvsadm -A -t 207.175.44.110:80 -s rr
444 ipvsadm -a -t 207.175.44.110:80 -r 192.168.10.1:80 -m
445 ipvsadm -a -t 207.175.44.110:80 -r 192.168.10.2:80 -m
446 ipvsadm -a -t 207.175.44.110:80 -r 192.168.10.3:80 -m
447 ipvsadm -a -t 207.175.44.110:80 -r 192.168.10.4:80 -m
448 ipvsadm -a -t 207.175.44.110:80 -r 192.168.10.5:80 -m
449 Alternatively, this could be achieved in a single ipvsadm command.
451 echo "
453 -A -t 207.175.44.110:80 -s rr
454 -a -t 207.175.44.110:80 -r 192.168.10.1:80 -m
455 -a -t 207.175.44.110:80 -r 192.168.10.2:80 -m
456 -a -t 207.175.44.110:80 -r 192.168.10.3:80 -m
457 -a -t 207.175.44.110:80 -r 192.168.10.4:80 -m
458 -a -t 207.175.44.110:80 -r 192.168.10.5:80 -m
459 " | ipvsadm -R
460 As masquerading is used as the forwarding mechanism in this example,
462 the default route of the real servers must be set to the linux direc‐
463 tor, which will need to be configured to forward and masquerade pack‐
464 ets. This can be achieved using the following commands:
465 echo "1" > /proc/sys/net/ipv4/ip_forward
467
469 The following commands configure a Linux Director to distribute incom‐
470 ing requests addressed to any port on 207.175.44.110 or 207.175.44.111
471 equally to the corresponding port on five real servers. As per the pre‐
472 vious example, the forwarding method used in this example is NAT, with
473 each of the real servers being masqueraded by the Linux Director.
474 ipvsadm -A -f 1 -s rr
476 ipvsadm -a -f 1 -r 192.168.10.1:0 -m
477 ipvsadm -a -f 1 -r 192.168.10.2:0 -m
478 ipvsadm -a -f 1 -r 192.168.10.3:0 -m
479 ipvsadm -a -f 1 -r 192.168.10.4:0 -m
480 ipvsadm -a -f 1 -r 192.168.10.5:0 -m
481 As masquerading is used as the forwarding mechanism in this example,
483 the default route of the real servers must be set to the linux direc‐
484 tor, which will need to be configured to forward and masquerade pack‐
485 ets. The real server should also be configured to mark incoming packets
486 addressed to any port on 207.175.44.110 and 207.175.44.111 with fire‐
487 wall-mark 1. If FTP traffic is to be handled by this virtual service,
488 then the ip_vs_ftp kernel module needs to be inserted into the kernel.
489 These operations can be achieved using the following commands:
490 echo "1" > /proc/sys/net/ipv4/ip_forward
492 modprobe ip_tables
493 iptables -A PREROUTING -t mangle -d 207.175.44.110/31 -j MARK --set-mark 1
494 modprobe ip_vs_ftp
495
497 IPv6 addresses should be surrounded by square brackets ([ and ]).
498 ipvsadm -A -t [2001:db8::80]:80 -s rr
500 ipvsadm -a -t [2001:db8::80]:80 -r [2001:db8::a0a0]:80 -m
501 fwmark IPv6 services require the -6 option.
503
505 The Linux Virtual Server implements three defense strategies against
506 some types of denial of service (DoS) attacks. The Linux Director cre‐
507 ates an entry for each connection in order to keep its state, and each
508 entry occupies 128 bytes effective memory. LVS's vulnerability to a DoS
509 attack lies in the potential to increase the number entries as much as
510 possible until the linux director runs out of memory. The three defense
511 strategies against the attack are: Randomly drop some entries in the
512 table. Drop 1/rate packets before forwarding them. And use secure tcp
513 state transition table and short timeouts. The strategies are con‐
514 trolled by sysctl variables and corresponding entries in the /proc
515 filesystem:
516 /proc/sys/net/ipv4/vs/drop_entry /proc/sys/net/ipv4/vs/drop_packet
518 /proc/sys/net/ipv4/vs/secure_tcp
519 Valid values for each variable are 0 through to 3. The default value is
521 0, which disables the respective defense strategy. 1 and 2 are auto‐
522 matic modes - when there is no enough available memory, the respective
523 strategy will be enabled and the variable is automatically set to 2,
524 otherwise the strategy is disabled and the variable is set to 1. A
525 value of 3 denotes that the respective strategy is always enabled. The
526 available memory threshold and secure TCP timeouts can be tuned using
527 the sysctl variables and corresponding entries in the /proc filesystem:
528 /proc/sys/net/ipv4/vs/amemthresh /proc/sys/net/ipv4/vs/timeout_*
530
532 /proc/net/ip_vs
533 /proc/net/ip_vs_app
534 /proc/net/ip_vs_conn
535 /proc/net/ip_vs_stats
536 /proc/sys/net/ipv4/vs/am_droprate
537 /proc/sys/net/ipv4/vs/amemthresh
538 /proc/sys/net/ipv4/vs/drop_entry
539 /proc/sys/net/ipv4/vs/drop_packet
540 /proc/sys/net/ipv4/vs/secure_tcp
541 /proc/sys/net/ipv4/vs/timeout_close
542 /proc/sys/net/ipv4/vs/timeout_closewait
543 /proc/sys/net/ipv4/vs/timeout_established
544 /proc/sys/net/ipv4/vs/timeout_finwait
545 /proc/sys/net/ipv4/vs/timeout_icmp
546 /proc/sys/net/ipv4/vs/timeout_lastack
547 /proc/sys/net/ipv4/vs/timeout_listen
548 /proc/sys/net/ipv4/vs/timeout_synack
549 /proc/sys/net/ipv4/vs/timeout_synrecv
550 /proc/sys/net/ipv4/vs/timeout_synsent
551 /proc/sys/net/ipv4/vs/timeout_timewait
552 /proc/sys/net/ipv4/vs/timeout_udp
553
555 The LVS web site (http://www.linuxvirtualserver.org/) for more documen‐
556 tation about LVS.
557 ipvsadm-save(8), ipvsadm-restore(8), iptables(8),
559 insmod(8), modprobe(8)
560
562 ipvsadm - Wensong Zhang <wensong@linuxvirtualserver.org>
563 Peter Kese <peter.kese@ijs.si>
564 man page - Mike Wangsmo <wanger@redhat.com>
565 Wensong Zhang <wensong@linuxvirtualserver.org>
566 Horms <horms@verge.net.au>
5674th Berkeley Distribution 5th July 2003 IPVSADM(8)