Advanced usage

Encrypting and decrypting from other programs

When using sops in scripts or from other programs, there are often situations where you do not want to write encrypted or decrypted data to disk. The best way to avoid this is to pass data to SOPS via stdin, and to let SOPS write data to stdout. By default, the encrypt and decrypt operations write data to stdout already. To pass data via stdin, you need to not provide an input filename. For encryption, you also must provide the --filename-override option with the file’s filename. The filename will be used to determine the input and output types, and to select the correct creation rule.

The simplest way to decrypt data from stdin is as follows:

$ cat encrypted-data | sops decrypt > decrypted-data

By default, sops determines the input and output format from the provided filename. Since in this case, no filename is provided, sops will use the binary store which expects JSON input and outputs binary data on decryption. This is often not what you want.

To avoid this, you can either provide a filename with --filename-override, or explicitly control the input and output formats by passing --input-type and --output-type as appropriate:

$ cat encrypted-data | sops decrypt --filename-override filename.yaml > decrypted-data
$ cat encrypted-data | sops decrypt --input-type yaml --output-type yaml > decrypted-data

In both cases, sops will assume that the data you provide is in YAML format, and will encode the decrypted data in YAML as well. The second form allows to use different formats for input and output.

To encrypt, it is important to note that SOPS also uses the filename to look up the correct creation rule from .sops.yaml. Therefore, you must provide the --filename-override parameter which allows you to tell SOPS which filename to use to match creation rules:

$ echo 'foo: bar' | sops encrypt --filename-override path/filename.sops.yaml > encrypted-data

SOPS will find a matching creation rule for path/filename.sops.yaml in .sops.yaml and use that one to encrypt the data from stdin. This filename will also be used to determine the input and output store. As always, the input store type can be adjusted by passing --input-type, and the output store type by passing --output-type:

$ echo foo=bar | sops encrypt --filename-override path/filename.sops.yaml --input-type dotenv > encrypted-data

Auditing

Sometimes, users want to be able to tell what files were accessed by whom in an environment they control. For this reason, SOPS can generate audit logs to record activity on encrypted files. When enabled, SOPS will write a log entry into a pre-configured PostgreSQL database when a file is decrypted. The log includes a timestamp, the username SOPS is running as, and the file that was decrypted.

In order to enable auditing, you must first create the database and credentials using the schema found in audit/schema.sql. This schema defines the tables that store the audit events and a role named sops that only has permission to add entries to the audit event tables. The default password for the role sops is sops. You should change this password.

Once you have created the database, you have to tell SOPS how to connect to it. Because we don't want users of SOPS to be able to control auditing, the audit configuration file location is not configurable, and must be at /etc/sops/audit.yaml. This file should have strict permissions such that only the root user can modify it.

For example, to enable auditing to a PostgreSQL database named sops running on localhost, using the user sops and the password sops, /etc/sops/audit.yaml should have the following contents:

backends:
    postgres:
        - connection_string: "postgres://sops:sops@localhost/sops?sslmode=verify-full"

You can find more information on the connection_string format in the PostgreSQL docs.

Under the postgres map entry in the above YAML is a list, so one can provide more than one backend, and SOPS will log to all of them:

backends:
    postgres:
        - connection_string: "postgres://sops:sops@localhost/sops?sslmode=verify-full"
        - connection_string: "postgres://sops:sops@remotehost/sops?sslmode=verify-full"

Passing Secrets to Other Processes

In addition to writing secrets to standard output and to files on disk, SOPS has two commands for passing decrypted secrets to a new process: exec-env and exec-file. These commands will place all output into the environment of a child process and into a temporary file, respectively. For example, if a program looks for credentials in its environment, exec-env can be used to ensure that the decrypted contents are available only to this process and never written to disk.

# print secrets to stdout to confirm values
$ sops decrypt out.json
{
        "database_password": "jf48t9wfw094gf4nhdf023r",
        "AWS_ACCESS_KEY_ID": "AKIAIOSFODNN7EXAMPLE",
        "AWS_SECRET_KEY": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

# decrypt out.json and run a command
# the command prints the environment variable and runs a script that uses it
$ sops exec-env out.json 'echo secret: $database_password; ./database-import'
secret: jf48t9wfw094gf4nhdf023r

# launch a shell with the secrets available in its environment
$ sops exec-env out.json 'sh'
sh-3.2# echo $database_password
jf48t9wfw094gf4nhdf023r

# the secret is not accessible anywhere else
sh-3.2$ exit
$ echo your password: $database_password
your password:

If the command you want to run only operates on files, you can use exec-file instead. By default, SOPS will use a FIFO to pass the contents of the decrypted file to the new program. Using a FIFO, secrets are only passed in memory which has two benefits: the plaintext secrets never touch the disk, and the child process can only read the secrets once. In contexts where this won't work, eg platforms like Windows where FIFOs unavailable or secret files that need to be available to the child process longer term, the --no-fifo flag can be used to instruct SOPS to use a traditional temporary file that will get cleaned up once the process is finished executing. exec-file behaves similar to find(1) in that {} is used as a placeholder in the command which will be substituted with the temporary file path (whether a FIFO or an actual file).

# operating on the same file as before, but as a file this time
$ sops exec-file out.json 'echo your temporary file: {}; cat {}'
your temporary file: /tmp/.sops894650499/tmp-file
{
        "database_password": "jf48t9wfw094gf4nhdf023r",
        "AWS_ACCESS_KEY_ID": "AKIAIOSFODNN7EXAMPLE",
        "AWS_SECRET_KEY": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}

# launch a shell with a variable TMPFILE pointing to the temporary file
$ sops exec-file --no-fifo out.json 'TMPFILE={} sh'
sh-3.2$ echo $TMPFILE
/tmp/.sops506055069/tmp-file291138648
sh-3.2$ cat $TMPFILE
{
        "database_password": "jf48t9wfw094gf4nhdf023r",
        "AWS_ACCESS_KEY_ID": "AKIAIOSFODNN7EXAMPLE",
        "AWS_SECRET_KEY": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
}
sh-3.2$ ./program --config $TMPFILE
sh-3.2$ exit

# try to open the temporary file from earlier
$ cat /tmp/.sops506055069/tmp-file291138648
cat: /tmp/.sops506055069/tmp-file291138648: No such file or directory

Additionally, on unix-like platforms, both exec-env and exec-file support dropping privileges before executing the new program via the --user <username> flag. This is particularly useful in cases where the encrypted file is only readable by root, but the target program does not need root privileges to function. This flag should be used where possible for added security.

To overwrite the default file name (tmp-file) in exec-file use the --filename <filename> parameter.

# the encrypted file can't be read by the current user
$ cat out.json
cat: out.json: Permission denied

# execute sops as root, decrypt secrets, then drop privileges
$ sudo sops exec-env --user nobody out.json 'sh'
sh-3.2$ echo $database_password
jf48t9wfw094gf4nhdf023r

# dropped privileges, still can't load the original file
sh-3.2$ id
uid=4294967294(nobody) gid=4294967294(nobody) groups=4294967294(nobody)
sh-3.2$ cat out.json
cat: out.json: Permission denied

Using SOPS’ Key Service

There are situations where you might want to run SOPS on a machine that doesn't have direct access to encryption keys such as PGP keys. The sops key service allows you to forward a socket so that SOPS can access encryption keys stored on a remote machine. This is similar to GPG Agent, but more portable.

SOPS uses a client-server approach to encrypting and decrypting the data key. By default, SOPS runs a local key service in-process. SOPS uses a key service client to send an encrypt or decrypt request to a key service, which then performs the operation. The requests are sent using gRPC and Protocol Buffers. The requests contain an identifier for the key they should perform the operation with, and the plaintext or encrypted data key. The requests do not contain any cryptographic keys, public or private.

WARNING: the key service connection currently does not use any sort of authentication or encryption. Therefore, it is recommended that you make sure the connection is authenticated and encrypted in some other way, for example through an SSH tunnel.

Whenever we try to encrypt or decrypt a data key, SOPS will try to do so first with the local key service (unless it's disabled), and if that fails, it will try all other remote key services until one succeeds.

You can start a key service server by running sops keyservice.

You can specify the key services the sops binary uses with the --keyservice option. This flag can be specified more than once, so you can use multiple key services. Alternatively, a single key service can be specified by setting the SOPS_KEYSERVICE environment variable. The local key service can be disabled with --enable-local-keyservice=false or by setting the SOPS_ENABLE_LOCAL_KEYSERVICE environment variable to false.

For example, to decrypt a file using both the local key service and the key service exposed on the unix socket located in /tmp/sops.sock, you can run:

$ sops decrypt --keyservice unix:///tmp/sops.sock file.yaml

And if you only want to use the key service exposed on the unix socket located in /tmp/sops.sock and not the local key service, you can run:

$ sops decrypt --enable-local-keyservice=false --keyservice unix:///tmp/sops.sock file.yaml