Introduction
This article will explain how to configure Squid and tweak a FreeBSD box for web caching and conserving bandwidth. Usually, Squid will reduce the traffic 30% or more from normal usage (without squid), and enhance response time.
This configuration and setting has been tested and the Squid FreeBSD boxes are running at several colleges in Sydney, Australia. Each Squid FreeBSD box serves about 100+ nodes.
Requirements
Squid needs a lot of memory. More is better but 128 MB of memory is good to start with. Squid also needs fast disk storage. Use SCSI drives if you can, though IDE drives will do the job. Also have a FreeBSD 4.5 box setup and running! You will need to recompile the kernel with additional options and components.
1. Installing Squid
You can install Squid using Ports but you can’t play with configure options, so I’ll cover the steps to install from a tarball.
You can download the Squid source from http://www.squid-cache.org. The lastest version is squid-2.4.STABLE6-src.tar.gz. Run the following commands as root.
# cd /path/to/tarball
# tar zxvf squid-2.4.STABLE6-src.tar.gz
# cd squid-2.4.STABLE6
# ./configure --enable-delay-pools --enable-ipf-transparent \
--enable-storeio=diskd,ufs --enable-storeio=diskd,ufs \
--disable-ident-lookups --enable-snmp --enable-removal-policies
# make all
# make install
The explaination of configure script options are below:
–enable-delay-pools - Enable delay pools to limit bandwidth usage.
You need to enable the option in order to use Squid to limit bandwith usage. It will give fair bandwith usage for everybody. In my case, I don’t want one person sucking all of the available bandwidth by downloading a big movie, causing others to suffer.
–enable-ipf-transparent - Enable Transparent Proxy support for systems using IP Filter network address redirection.
With this option, you don’t have to configure the client’s browser proxy setting. Also it is a good way to force the client to use the proxy everytime.
–enable-storeio=diskd,ufs - Enable diskd
Improve disk I/O performance. According to the Squid FAQ, if you enable diskd you can gain a 400% increase of perfomance. However, you would need to recompile the kernel because your operating system must support message queues and shared memory.
–enable-removal-policies - Build support for the list of removal policies.
By default, Squid uses LRU, but there are two better policies: GDSF and LFUDA. See the Squid config for a more detailed explanation.
–disable-ident-lookups - This allows you to remove code that performs Ident (RFC 931) lookups.
Not really important. By the way, if you do transparent proxy, ident lookups won’t work.
–enable-snmp
Optional: enable this and you can monitor Squid with mrtg or rrdtool. How to do this is outside of this article’s scope. Perhaps in my next one.
2. Edit Squid Configuration File /usr/local/squid/etc/squid.conf
# Need for transparent proxy
# You need to --enable-ipf-transparent
http_port 3128
httpd_accel_host virtual
httpd_accel_port 80
httpd_accel_with_proxy on
httpd_accel_uses_host_header on
# Physical memory / 3
cache_mem 128 MB
# Max out Squid I/O perfomance, 15 GB cache and use Squid special diskd but you need to recompile the kernel
# To use disk you need to --enable-storeio=diskd,ufs
# Reasonable values for Q1 and Q2 are 72 and 64, respectively.
# Q1 value must bigger Q2
cache_dir diskd /usr/local/squid/cache 15360 16 256 Q1=72 Q2=64
# You can use normal ufs instead
#cache_dir ufs /usr/local/squid/cache 15360 16 256
# I dont want to log anything
# The reason is to save some expensive I/O operation.
cache_access_log /dev/null
cache_store_log none
cache_log /dev/null
# Cache replacement policy
# The heap GDSF policy optimizes object-hit rate by keeping smaller popular
# objects in cache, so it has a better chance of getting a hit. It achieves a
# lower byte hit rate than LFUDA, though, since it evicts larger (possibly popular)
# objects.
# The heap LFUDA ( Least Frequently Used with Dynamic Aging ) policy keeps
# popular objects in cache regardless of their size and thus optimizes byte hit
# rate at the expense of hit rate since one large, popular object will prevent
# many smaller, slightly less popular objects from being cached.
# You need to --enable-removal-policies
cache_replacement_policy GDSF
# Standard Access List
# I have two subnets, one for student and another one for admin
# Modify this according to your network
acl all src 0.0.0.0/0.0.0.0
acl manager proto cache_object
acl localhost src 127.0.0.1/255.255.255.255
acl outgoing src 192.168.10.2/255.255.255.255
acl student src 192.168.0.0/255.255.255.0
acl admin src 192.168.1.0/255.255.255.0
acl SSL_ports port 443 563
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 563 # https, snews
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl CONNECT method CONNECT
http_access allow manager
http_access allow localhost
http_access allow outgoing
http_access allow student
http_access allow admin
http_access deny !Safe_ports
http_access deny CONNECT !SSL_ports
http_access deny all
icp_access allow localhost
icp_access allow student
icp_access allow admin
icp_access deny all
# Avoid caching cgi scripts
acl QUERY urlpath_regex cgi-bin
no_cache deny QUERY
acl magic_words1 url_regex -i 192.168
acl magic_words2 url_regex -i ftp .exe .mp3 .vqf .tar.gz .gz .rpm .zip .rar .avi .mpeg .mpe .mpg .qt .ram .rm .iso .raw .wav .mov
# Delay Pool
# For delay pool, you need to --enable-delay-pools
delay_pools 2
# I have ADSL 2Mbits line
# 2 mbits == 256 kbytes per second
# 256 KB/s, 5 KB/s
# It means 256 KB/s bandwith for the whole network, but 5 KB/s for each node, which is fair for everybody
delay_class 1 2
delay_parameters 1 256000/256000 5000/256000
delay_access 1 allow magic_words2
delay_access 1 allow student
delay_access 1 allow admin
# -1/-1 means that there are no limits for local traffic.
delay_class 2 2
delay_parameters 2 -1/-1 -1/-1
delay_access 2 allow magic_words1
# Cancel download if file is bigger than 1MB
reply_body_max_size 1024 KB
# snmp stuff
acl snmppublic snmp_community public
snmp_access allow snmppublic localhost
snmp_access deny all
# Change to your domain
# visible_hostname yourdomain.domain.com
# cache_mgr yourname@youremail.com
3. Create cache dir and create swap
# mkdir /usr/local/squid/cache
# chown nobody:nogroup cache
# /usr/local/squid/bin/squid -k parse
# /usr/local/squid/bin/squid -z
4. Configure transparent proxy with ipfilter
4.1 Edit /etc/rc.conf
# add these lines to enable ipfilter
ipfilter_enable="YES"
ipnat_enable="YES"
ipmon_enable="YES"
ipfs_enable="YES"
4.2 Edit /etc/ipnat.rules
# add this line
# I assume rl0 is your internal nic
# Redirect everything else to squid on port 3128
rdr rl0 0/0 port 80 -> 127.0.0.1 port 3128 tcp
5. Reconfigure kernel for squid diskd support
Consult the Freebsd Handbook for recompiling the kernel and add the following lines. Your kernel must have:
options SYSVMSG
You can set the parameters in the kernel as follows. This is just an example. Make sure the values are appropriate for your system:
options MSGMNB=8192 # max # of bytes in a queue
options MSGMNI=40 # number of message queue identifiers
options MSGSEG=512 # number of message segments per queue
options MSGSSZ=64 # size of a message segment
options MSGTQL=2048 # max messages in system
The following is the explanation of the kernel options from the Squid FAQ:
The messages between Squid and diskd are 32 bytes for 32-bit CPUs and 40 bytes for 64-bit CPUs. Thus, MSGSSZ should be 32 or greater. You may want to set it to a larger value, just to be safe.
We’ll have two queues for each cache_dir, one in each direction. So, MSGMNI needs to be at least two times the number of cache_dir’s.
I’ve found that 75 messages per queue is about the limit of decent performance. If each diskd message consists of just one segment (depending on your value of MSGSSZ), then MSGSEG should be greater than 75.
MSGMNB and MSGTQL affect how many messages can be in the queues at one time. Diskd messages shouldn’t be more than 40 bytes, but let’s use 64 bytes to be safe. MSGMNB should be at least 64*75. I recommend rounding up to the nearest power of two, or 8192.
MSGTQL should be at least 75 times the number of cache_dir’s that you’ll have.
Also you can tweak the kernel by commenting out unnecessary lines in the kernel config to gain extra perfomance. Then recompile the kernel.
6. Create start-up script /usr/local/etc/rc.d/squid.sh
#!/bin/sh
echo -n ' Squid '
case "$1" in
start)
/usr/local/squid/bin/squid -D
;;
stop)
/usr/local/squid/bin/squid -k shutdown
;;
restart)
/usr/local/squid/bin/squid -k reconfigure
;;
*)
echo "Usage: `basename $0` {start|stop|restart}"
;;
esac
7. All Done!
