1IPVSADM(8)                Linux Administrator's Guide               IPVSADM(8)
2
3
4

NAME

6       ipvsadm - Linux Virtual Server administration
7

SYNOPSIS

9       ipvsadm -A|E virtual-service [-s scheduler]
10               [-p [timeout]] [-M netmask] [-b sched-flags]
11       ipvsadm -D virtual-service
12       ipvsadm -C
13       ipvsadm -R
14       ipvsadm -S [-n]
15       ipvsadm -a|e virtual-service -r server-address
16               [-g|i|m] [-w weight] [-x upper] [-y lower]
17       ipvsadm -d virtual-service -r server-address
18       ipvsadm -L|l [virtual-service] [options]
19       ipvsadm -Z [virtual-service]
20       ipvsadm --set tcp tcpfin udp
21       ipvsadm --start-daemon state [daemon-options]
22               [--syncid syncid]
23       ipvsadm --stop-daemon state
24       ipvsadm -h
25

DESCRIPTION

27       Ipvsadm(8)  is  used  to set up, maintain or inspect the virtual server
28       table in the Linux kernel. The Linux Virtual  Server  can  be  used  to
29       build  scalable  network  services  based  on  a cluster of two or more
30       nodes. The active node of the cluster redirects service requests  to  a
31       collection  of  server  hosts  that will actually perform the services.
32       Supported features include three protocols (TCP, UDP and  SCTP),  three
33       packet-forwarding  methods  (NAT,  tunneling,  and direct routing), and
34       eight load balancing algorithms (round  robin,  weighted  round  robin,
35       least-connection,  weighted least-connection, locality-based least-con‐
36       nection, locality-based least-connection with replication, destination-
37       hashing, and source-hashing).
38
39       The command has two basic formats for execution:
40
41       ipvsadm COMMAND virtual-service
42               [scheduling-method] [persistence options]
43
44       ipvsadm command virtual-service
45               server-address [packet-forwarding-method]
46               [weight options]
47
48       The  first  format  manipulates a virtual service and the algorithm for
49       assigning service requests to real servers.  Optionally,  a  persistent
50       timeout  and  network  mask for the granularity of a persistent service
51       may be specified. The second format manipulates a real server  that  is
52       associated  with  an  existing  virtual service. When specifying a real
53       server, the packet-forwarding method and the weight of the real server,
54       relative  to  other real servers for the virtual service, may be speci‐
55       fied, otherwise defaults will be used.
56
57   COMMANDS
58       ipvsadm(8) recognises the commands described below. Upper-case commands
59       maintain  virtual  services.  Lower-case commands maintain real servers
60       that are associated with a virtual service.
61
62       -A, --add-service
63              Add a virtual service. A service address is uniquely defined  by
64              a triplet: IP address, port number, and protocol. Alternatively,
65              a virtual service may be defined by a firewall-mark.
66
67       -E, --edit-service
68              Edit a virtual service.
69
70       -D, --delete-service
71              Delete  a  virtual  service,  along  with  any  associated  real
72              servers.
73
74       -C, --clear
75              Clear the virtual server table.
76
77       -R, --restore
78              Restore  Linux  Virtual  Server rules from stdin. Each line read
79              from stdin will be treated as the command line options to a sep‐
80              arate  invocation  of ipvsadm. Lines read from stdin can option‐
81              ally begin with "ipvsadm".  This option is useful to avoid  exe‐
82              cuting  a large number or ipvsadm  commands when constructing an
83              extensive routing table.
84
85       -S, --save
86              Dump the Linux Virtual Server rules to stdout in a  format  that
87              can be read by -R|--restore.
88
89       -a, --add-server
90              Add a real server to a virtual service.
91
92       -e, --edit-server
93              Edit a real server in a virtual service.
94
95       -d, --delete-server
96              Remove a real server from a virtual service.
97
98       -L, -l, --list
99              List  the virtual server table if no argument is specified. If a
100              service-address is selected, list this service only. If  the  -c
101              option is selected, then display the connection table. The exact
102              output is affected by the other arguments given.
103
104       -Z, --zero
105              Zero the packet, byte and rate counters in a service or all ser‐
106              vices.
107
108       --set tcp tcpfin udp
109              Change  the  timeout values used for IPVS connections. This com‐
110              mand always takes  3  parameters,   representing   the   timeout
111              values (in seconds) for TCP sessions, TCP sessions after receiv‐
112              ing a  FIN packet, and  UDP  packets, respectively.   A  timeout
113              value 0 means that the current timeout value of the  correspond‐
114              ing  entry  is preserved.
115
116       --start-daemon state
117              Start the connection synchronization daemon.  The  state  is  to
118              indicate  that  the  daemon  is started as master or backup. The
119              connection synchronization  daemon  is  implemented  inside  the
120              Linux kernel. The master daemon running at the primary load bal‐
121              ancer multicasts changes of connections  periodically,  and  the
122              backup daemon running at the backup load balancers receives mul‐
123              ticast message and creates corresponding connections.  Then,  in
124              case  the  primary  load  balancer fails, a backup load balancer
125              will takeover, and it has state of almost  all  connections,  so
126              that  almost  all established connections can continue to access
127              the service.
128
129       The sync daemon supports IPv4 and IPv6 connections.
130
131       --stop-daemon
132              Stop the connection synchronization daemon.
133
134       -h, --help
135              Display a description of the command syntax.
136
137   virtual-service
138       Specifies the virtual service based on protocol/addr/port  or  firewall
139       mark.
140
141       -t, --tcp-service service-address
142              Use TCP service. The service-address is of the form host[:port].
143              Host may be one of a plain IP address or a hostname. Port may be
144              either a plain port number or the service name of port. The Port
145              may be omitted, in which case zero will be used. A Port  of zero
146              is  only  valid if the service is persistent as the -p|--persis‐
147              tent option, in which case it is a wild-card port, that is  con‐
148              nections will be accepted to any port.
149
150       -u, --udp-service service-address
151              Use UDP service. See the -t|--tcp-service for the description of
152              the service-address.
153
154       --sctp-service service-address
155              Use SCTP service. See the -t|--tcp-service for  the  description
156              of the service-address.
157
158       -f, --fwmark-service integer
159              Use  a  firewall-mark,  an  integer  value greater than zero, to
160              denote a virtual service instead of an address, port and  proto‐
161              col (UDP or TCP). The marking of packets with a firewall-mark is
162              configured using the -m|--mark option to iptables(8). It can  be
163              used  to  build  a virtual service associated with the same real
164              servers,  covering  multiple  IP  address,  port  and   protocol
165              triplets.  If  IPv6  addresses  are  used, the -6 option must be
166              used.
167
168              Using  firewall-mark  virtual  services  provides  a  convenient
169              method  of  grouping  together different IP addresses, ports and
170              protocols into a single virtual service. This is useful for both
171              simplifying  configuration if a large number of virtual services
172              are required and grouping persistence across what  would  other‐
173              wise be multiple virtual services.
174
175   PARAMETERS
176       The  commands  above  accept  or  require zero or more of the following
177       parameters.
178
179       -s, --scheduler scheduling-method
180              scheduling-method  Algorithm for allocating TCP connections  and
181              UDP datagrams to real servers.  Scheduling algorithms are imple‐
182              mented as kernel modules. Ten are shipped with the Linux Virtual
183              Server:
184
185              rr - Round Robin: distributes jobs equally amongst the available
186              real servers.
187
188              wrr - Weighted Round Robin: assigns jobs to real servers propor‐
189              tionally  to  there  real  servers'  weight. Servers with higher
190              weights receive new jobs first and get more  jobs  than  servers
191              with lower weights. Servers with equal weights get an equal dis‐
192              tribution of new jobs.
193
194              lc - Least-Connection: assigns more jobs to  real  servers  with
195              fewer active jobs.
196
197              wlc  -  Weighted  Least-Connection: assigns more jobs to servers
198              with fewer  jobs  and  relative  to  the  real  servers'  weight
199              (Ci/Wi). This is the default.
200
201              lblc  -  Locality-Based  Least-Connection: assigns jobs destined
202              for the same IP address to the same server if the server is  not
203              overloaded  and available; otherwise assign jobs to servers with
204              fewer jobs, and keep it for future assignment.
205
206              lblcr  -  Locality-Based  Least-Connection   with   Replication:
207              assigns  jobs destined for the same IP address to the least-con‐
208              nection node in the server set for the IP address.  If  all  the
209              node  in the server set are over loaded, it picks up a node with
210              fewer jobs in the cluster and adds it in the sever set  for  the
211              target.  If  the server set has not been modified for the speci‐
212              fied time, the most loaded node is removed from the server  set,
213              in order to avoid high degree of replication.
214
215              dh  - Destination Hashing: assigns jobs to servers through look‐
216              ing up a statically assigned hash table by their destination  IP
217              addresses.
218
219              sh  - Source Hashing: assigns jobs to servers through looking up
220              a statically assigned hash table by their source  IP  addresses.
221              This  scheduler  has two flags: sh-fallback, which enables fall‐
222              back to a different server if the selected server  was  unavail‐
223              able, and sh-port, which adds the source port number to the hash
224              computation.
225
226              sed - Shortest Expected Delay: assigns an incoming  job  to  the
227              server with the shortest expected delay. The expected delay that
228              the job will experience is (Ci + 1) / Ui if   sent  to  the  ith
229              server,  in which Ci is the number of jobs on the the ith server
230              and Ui is the fixed service rate (weight) of the ith server.
231
232              nq - Never Queue: assigns an incoming job to an idle  server  if
233              there  is, instead of waiting for a fast one; if all the servers
234              are busy, it adopts the Shortest Expected Delay policy to assign
235              the job.
236
237       -p, --persistent [timeout]
238              Specify  that a virtual service is persistent. If this option is
239              specified, multiple requests from a client are redirected to the
240              same  real  server  selected for the first request.  Optionally,
241              the timeout of persistent sessions may  be  specified  given  in
242              seconds, otherwise the default of 300 seconds will be used. This
243              option may be used in conjunction with protocols such as SSL  or
244              FTP where it is important that clients consistently connect with
245              the same real server.
246
247              Note: If a virtual service is to  handle  FTP  connections  then
248              persistence  must be set for the virtual service if Direct Rout‐
249              ing or Tunnelling is used as the forwarding mechanism.  If  Mas‐
250              querading  is  used in conjunction with an FTP service than per‐
251              sistence is not necessary, but the ip_vs_ftp kernel module  must
252              be  used.   This module may be manually inserted into the kernel
253              using insmod(8).
254
255       -M, --netmask netmask
256              Specify the granularity with which clients are grouped for  per‐
257              sistent  virtual services.  The source address of the request is
258              masked with this netmask to direct all clients from a network to
259              the  same  real server. The default is 255.255.255.255, that is,
260              the persistence granularity is per client  host.  Less  specific
261              netmasks  may  be  used  to resolve problems with non-persistent
262              cache clusters on the client  side.   IPv6  netmasks  should  be
263              specified  as  a  prefix  length between 1 and 128.  The default
264              prefix length is 128.
265
266       -b, --sched-flags sched-flags
267              Set scheduler flags for this virtual server.  sched-flags  is  a
268              comma-separated  list  of flags.  See the scheduler descriptions
269              for valid scheduler flags.
270
271       -r, --real-server server-address
272              Real server that  an  associated  request  for  service  may  be
273              assigned  to.   The server-address is the host address of a real
274              server, and may plus port. Host can be either a plain IP address
275              or  a  hostname.   Port can be either a plain port number or the
276              service name of port.  In the case of the  masquerading  method,
277              the  host address is usually an RFC 1918 private IP address, and
278              the port can be different from that of the  associated  service.
279              With  the  tunneling  and  direct  routing methods, port must be
280              equal to that of the service address. For normal  services,  the
281              port  specified   in the service address will be used if port is
282              not specified. For fwmark services,  port  may  be  omitted,  in
283              which  case  the destination port on the real server will be the
284              destination port of the request sent to the virtual service.
285
286       [packet-forwarding-method]
287
288              -g, --gatewaying  Use gatewaying (direct routing). This  is  the
289              default.
290
291              -i, --ipip  Use ipip encapsulation (tunneling).
292
293              -m,  --masquerading   Use  masquerading (network access transla‐
294              tion, or NAT).
295
296              Note:  Regardless of the packet-forwarding mechanism  specified,
297              real servers for addresses for which there are interfaces on the
298              local node will be use the local forwarding method, then packets
299              for the servers will be passed to upper layer on the local node.
300              This cannot be specified by ipvsadm, rather it set by the kernel
301              as real servers are added or modified.
302
303       -w, --weight weight
304              Weight  is an integer specifying the capacity  of a server rela‐
305              tive to the others in the pool. The valid values of weight are 0
306              through to 65535. The default is 1. Quiescent servers are speci‐
307              fied with a weight of zero. A quiescent server will  receive  no
308              new  jobs  but still serve the existing jobs, for all scheduling
309              algorithms distributed with the Linux Virtual Server. Setting  a
310              quiescent  server  may  be useful if the server is overloaded or
311              needs to be taken out of service for maintenance.
312
313       -x, --u-threshold uthreshold
314              uthreshold is an integer specifying the upper connection thresh‐
315              old of a server. The valid values of uthreshold are 0 through to
316              65535. The default  is  0,  which  means  the  upper  connection
317              threshold is not set. If uthreshold is set with other values, no
318              new connections will be sent to the server when  the  number  of
319              its connections exceeds its upper connection threshold.
320
321       -y, --l-threshold lthreshold
322              lthreshold is an integer specifying the lower connection thresh‐
323              old of a server. The valid values of lthreshold are 0 through to
324              65535.  The  default  is  0,  which  means  the lower connection
325              threshold is not set. If lthreshold is set  with  other  values,
326              the  server  will receive new connections when the number of its
327              connections drops  below  its  lower  connection  threshold.  If
328              lthreshold  is  not  set  but uthreshold is set, the server will
329              receive new connections when the number of its connections drops
330              below three forth of its upper connection threshold.
331
332       -c, --connection
333              Connection  output.  The list command with this option will list
334              current IPVS connections.
335
336       --timeout
337              Timeout output. The list command with this option  will  display
338              the   timeout values (in seconds) for TCP sessions, TCP sessions
339              after receiving a FIN packet, and UDP packets.
340
341       --daemon
342              Daemon information output. The list  command  with  this  option
343              will display the daemon status and its multicast interface.
344
345       --stats
346              Output  of  statistics  information.  The list command with this
347              option will display the statistics information of  services  and
348              their servers.
349
350       --rate Output  of  rate  information. The list command with this option
351              will display the rate information (such  as  connections/second,
352              bytes/second and packets/second) of services and their servers.
353
354       --thresholds
355              Output  of  thresholds  information.  The list command with this
356              option will display the upper/lower connection threshold  infor‐
357              mation of each server in service listing.
358
359       --persistent-conn
360              Output  of  persistent  connection information. The list command
361              with this option will display the persistent connection  counter
362              information  of  each  server in service listing. The persistent
363              connection is used to forward the actual  connections  from  the
364              same client/network to the same server.
365
366              The  list  command  with  the  -c,  --connection option and this
367              option will include persistence engine data, if any is  present,
368              when listing connections.
369
370       --sort Sort  the list of virtual services and real servers. The virtual
371              service entries are sorted  in  ascending  order  by  <protocol,
372              address,  port>. The real server entries are sorted in ascending
373              order by <address, port>. (default)
374
375       --nosort
376              Do not sort the list of virtual services and real servers.
377
378       -n, --numeric
379              Numeric output.  IP addresses and port numbers will  be  printed
380              in  numeric  format  rather  than  as as host names and services
381              respectively, which is the  default.
382
383       --exact
384              Expand numbers.  Display the exact value of the packet and  byte
385              counters,  instead  of only the rounded number in K's (multiples
386              of 1000) M's (multiples of 1000K) or G's (multiples  of  1000M).
387              This option is only relevant for the -L command.
388
389       -6, --ipv6
390              Use with -f to signify fwmark rule uses IPv6 addresses.
391
392       -o, --ops
393              One-packet  scheduling.   Used in conjunction with a UDP virtual
394              service or a fwmark virtual service that handles only UDP  pack‐
395              ets.   All  connections are created such that they only schedule
396              one packet.
397
398   PARAMETERS FOR SYNCHRONIZATION DAEMON
399       The --start-daemon requires zero or more of the following parameters.
400
401       --syncid syncid
402              Specify the syncid that the sync master daemon fills in the Syn‐
403              cID  header while sending multicast messages, or the sync backup
404              daemon uses to filter out multicast messages  not  matched  with
405              the  SyncID  value.  The valid values of syncid are 0 through to
406              255. The default is 0, which means no filtering at all.
407
408       --sync-maxlen length
409              Specify the desired length of sync messages (UDP payload  size).
410              It  is  expected that backup server will use value not less than
411              the used value in master server.  The valid values of length are
412              in  the  1  ..  (65535  - 20 - 8) range but the kernel ensures a
413              space for at least one sync message.  If value is lower than MTU
414              the  sync  messages will be fragmented by IP layer.  The default
415              value is derived from the MTU value when daemon is  started  but
416              master  daemon will not default to value above 1500 for compati‐
417              bility reasons.
418
419       --mcast-interface interface
420              Specify the multicast interface  that  the  sync  master  daemon
421              sends  outgoing  multicasts  through,  or the sync backup daemon
422              listens to for multicasts.
423
424       --mcast-group address
425              Specify IPv4 or IPv6 multicast address for  the  sync  messages.
426              The default value is 224.0.0.81.
427
428       --mcast-port port
429              Specify  the  UDP  port for sync messages.  The default value is
430              8848.
431
432       --mcast-ttl ttl
433              Specify the TTL value for sync messages (1 .. 255).  The default
434              value is 1.
435

EXAMPLE 1 - Simple Virtual Service

437       The  following commands configure a Linux Director to distribute incom‐
438       ing requests addressed to port 80 on 207.175.44.110 equally to port  80
439       on  five  real  servers.  The forwarding method used in this example is
440       NAT, with each of the real  servers  being  masqueraded  by  the  Linux
441       Director.
442
443       ipvsadm -A -t 207.175.44.110:80 -s rr
444       ipvsadm -a -t 207.175.44.110:80 -r 192.168.10.1:80 -m
445       ipvsadm -a -t 207.175.44.110:80 -r 192.168.10.2:80 -m
446       ipvsadm -a -t 207.175.44.110:80 -r 192.168.10.3:80 -m
447       ipvsadm -a -t 207.175.44.110:80 -r 192.168.10.4:80 -m
448       ipvsadm -a -t 207.175.44.110:80 -r 192.168.10.5:80 -m
449
450       Alternatively, this could be achieved in a single ipvsadm command.
451
452       echo "
453       -A -t 207.175.44.110:80 -s rr
454       -a -t 207.175.44.110:80 -r 192.168.10.1:80 -m
455       -a -t 207.175.44.110:80 -r 192.168.10.2:80 -m
456       -a -t 207.175.44.110:80 -r 192.168.10.3:80 -m
457       -a -t 207.175.44.110:80 -r 192.168.10.4:80 -m
458       -a -t 207.175.44.110:80 -r 192.168.10.5:80 -m
459       " | ipvsadm -R
460
461       As  masquerading  is  used as the forwarding mechanism in this example,
462       the default route of the real servers must be set to the  linux  direc‐
463       tor,  which  will need to be configured to forward and masquerade pack‐
464       ets. This can be achieved using the following commands:
465
466       echo "1" > /proc/sys/net/ipv4/ip_forward
467

EXAMPLE 2 - Firewall-Mark Virtual Service

469       The following commands configure a Linux Director to distribute  incom‐
470       ing  requests addressed to any port on 207.175.44.110 or 207.175.44.111
471       equally to the corresponding port on five real servers. As per the pre‐
472       vious  example, the forwarding method used in this example is NAT, with
473       each of the real servers being masqueraded by the Linux Director.
474
475       ipvsadm -A -f 1  -s rr
476       ipvsadm -a -f 1 -r 192.168.10.1:0 -m
477       ipvsadm -a -f 1 -r 192.168.10.2:0 -m
478       ipvsadm -a -f 1 -r 192.168.10.3:0 -m
479       ipvsadm -a -f 1 -r 192.168.10.4:0 -m
480       ipvsadm -a -f 1 -r 192.168.10.5:0 -m
481
482       As masquerading is used as the forwarding mechanism  in  this  example,
483       the  default  route of the real servers must be set to the linux direc‐
484       tor, which will need to be configured to forward and  masquerade  pack‐
485       ets. The real server should also be configured to mark incoming packets
486       addressed to any port on 207.175.44.110 and  207.175.44.111 with  fire‐
487       wall-mark  1.  If FTP traffic is to be handled by this virtual service,
488       then the ip_vs_ftp kernel module needs to be inserted into the  kernel.
489       These operations can be achieved using the following commands:
490
491       echo "1" > /proc/sys/net/ipv4/ip_forward
492       modprobe ip_tables
493       iptables  -A PREROUTING -t mangle -d 207.175.44.110/31 -j MARK --set-mark 1
494       modprobe ip_vs_ftp
495

IPv6

497       IPv6 addresses should be surrounded by square brackets ([ and ]).
498
499       ipvsadm -A -t [2001:db8::80]:80 -s rr
500       ipvsadm -a -t [2001:db8::80]:80 -r [2001:db8::a0a0]:80 -m
501
502       fwmark IPv6 services require the -6 option.
503

NOTES

505       The  Linux  Virtual  Server implements three defense strategies against
506       some types of denial of service (DoS) attacks. The Linux Director  cre‐
507       ates  an entry for each connection in order to keep its state, and each
508       entry occupies 128 bytes effective memory. LVS's vulnerability to a DoS
509       attack  lies in the potential to increase the number entries as much as
510       possible until the linux director runs out of memory. The three defense
511       strategies  against  the  attack are: Randomly drop some entries in the
512       table. Drop 1/rate packets before forwarding them. And use  secure  tcp
513       state  transition  table  and  short  timeouts. The strategies are con‐
514       trolled by sysctl variables and  corresponding  entries  in  the  /proc
515       filesystem:
516
517       /proc/sys/net/ipv4/vs/drop_entry      /proc/sys/net/ipv4/vs/drop_packet
518       /proc/sys/net/ipv4/vs/secure_tcp
519
520       Valid values for each variable are 0 through to 3. The default value is
521       0,  which  disables  the respective defense strategy. 1 and 2 are auto‐
522       matic modes - when there is no enough available memory, the  respective
523       strategy  will  be  enabled and the variable is automatically set to 2,
524       otherwise the strategy is disabled and the variable  is  set  to  1.  A
525       value of 3 denotes that the respective strategy is always enabled.  The
526       available memory threshold and secure TCP timeouts can be  tuned  using
527       the sysctl variables and corresponding entries in the /proc filesystem:
528
529       /proc/sys/net/ipv4/vs/amemthresh /proc/sys/net/ipv4/vs/timeout_*
530

FILES

532       /proc/net/ip_vs
533       /proc/net/ip_vs_app
534       /proc/net/ip_vs_conn
535       /proc/net/ip_vs_stats
536       /proc/sys/net/ipv4/vs/am_droprate
537       /proc/sys/net/ipv4/vs/amemthresh
538       /proc/sys/net/ipv4/vs/drop_entry
539       /proc/sys/net/ipv4/vs/drop_packet
540       /proc/sys/net/ipv4/vs/secure_tcp
541       /proc/sys/net/ipv4/vs/timeout_close
542       /proc/sys/net/ipv4/vs/timeout_closewait
543       /proc/sys/net/ipv4/vs/timeout_established
544       /proc/sys/net/ipv4/vs/timeout_finwait
545       /proc/sys/net/ipv4/vs/timeout_icmp
546       /proc/sys/net/ipv4/vs/timeout_lastack
547       /proc/sys/net/ipv4/vs/timeout_listen
548       /proc/sys/net/ipv4/vs/timeout_synack
549       /proc/sys/net/ipv4/vs/timeout_synrecv
550       /proc/sys/net/ipv4/vs/timeout_synsent
551       /proc/sys/net/ipv4/vs/timeout_timewait
552       /proc/sys/net/ipv4/vs/timeout_udp
553

SEE ALSO

555       The LVS web site (http://www.linuxvirtualserver.org/) for more documen‐
556       tation about LVS.
557
558       ipvsadm-save(8), ipvsadm-restore(8), iptables(8),
559       insmod(8), modprobe(8)
560

AUTHORS

562       ipvsadm - Wensong Zhang <wensong@linuxvirtualserver.org>
563              Peter Kese <peter.kese@ijs.si>
564       man page - Mike Wangsmo <wanger@redhat.com>
565               Wensong Zhang <wensong@linuxvirtualserver.org>
566               Horms <horms@verge.net.au>
567
568
569
5704th Berkeley Distribution        5th July 2003                      IPVSADM(8)
Impressum