Commit 97f47a01 authored by ∞'s avatar 💻

Bigsudo release

parent a8980440
Pipeline #2827 failed with stages
in 29 seconds
stages:
- test
- release
variables:
ANSIBLE_FORCE_COLOR: 'true'
py-sec-safety:
stage: test
image: yourlabs/python
script: safety check
py-qa:
stage: test
image: yourlabs/python
script: flake8 bigsudo
.test: &test
stage: test
image: yourlabs/python
before_script:
- pip install --upgrade --editable .
- set -eux
test-playbook-executable:
<<: *test
script: |
./example/playbook.yml
grep hello ./playbook.out
test-clones-main:
<<: *test
script: |
bigsudo -v yourlabs.io/oss/yourlabs.bigsudo-example @localhost example_variable=$(pwd)/test1
grep main test1
test-clone-update:
<<: *test
script: |
bigsudo -v yourlabs.io/oss/yourlabs.bigsudo-example @localhost example_variable=$(pwd)/test2 update
grep update test2
test-clone-local:
<<: *test
script: |
git clone https://yourlabs.io/oss/yourlabs.bigsudo-example.git /example
bigsudo -v /example @localhost example_variable=$(pwd)/test3
grep main test3
bigsudo -v /example @localhost example_variable=$(pwd)/test4 update
grep update test4
pypi:
stage: deploy
stage: release
image: yourlabs/python
except: [tags]
script:
- shyml twine | bash -eux
script: pypi-release
ansible-apply: apply remote tasks, supports command line inventory
==================================================================
bigsudo: Obscene ansible runner
===============================
Ansible-apply takes the tasks file spec as first argument::
Bigsudo is an opinionated command line wrapper to ansible-playbook.
ansible-apply role.name
ansible-apply role.name/some_tasks
ansible-apply https://some/playbook.yml
ansible-apply ./playbook.yml
It accepts as first argument: role name, path or url, or playbook path
or url::
It will automatically download the playbook or role if not found.
bigsudo role.name # download role and run tasks/main.yml on localhost
Then, the command takes any number of hosts and inventory variables on
the command line::
bigsudo role.name update # do tasks/update.yml
bigsudo role.name user@host update # do tasks/update.yml on host
bigsudo role.name @host update # with current user
bigsudo role.name @host update foo=bar # custom variable
bigsudo role.name {"foo":"bar"} # also accepts json without space
bigsudo role.name --become # forwards any ansible-playbook argument
ansible-apply role.name server1 server2 update=true
Note that bigsudo will automatically call ansible-galaxy install on
requirements.yml it finds in any role, recursively on each role that it got
galaxy to install. This means that yourlabs.docker/requirements.yml will also
be installed by bigsudo if your repo has this requirements.yml::
Finnaly, any argument passed with dashes are forwarded to the
ansible-playbook command it generates, but named args must use the ``=``
notation, and not a space to not confuse the command line parser::
- src: git+https://yourlabs.io/oss/yourlabs.docker
# works:
ansible-apply role.name server2 update=true --become-user=root
# does not:
ansible-apply role.name server2 update=true --become-user root
The gotcha is that you cannot pass values to a short-written argument (because
it's my opinion that ansible commands are more readable as such), ie::
# works:
$ ./example/playbook.yml --tags=foo
ansible-playbook --tags=foo -c local -i localhost, -e apply_tasks='["main"]' ./example/playbook.yml
# does NOT work: parser doesn't detect that foo is the value of -t:
$ ./example/playbook.yml -t foo
ansible-playbook -t -c local -i localhost, -e apply_tasks='["foo"]' ./example/playbook.yml
# does NOT work: parser doesn't detect that foo is the value of --tags:
$ ./example/playbook.yml --tags foo
ansible-playbook --tags -c local -i localhost, -e apply_tasks='["foo"]' ./example/playbook.yml
Using gitlab-ci you can define multiline env vars, ie a with
$STAGING_HOST=deploy@yourstaging and json string for $STAGING_VARS::
{
"security_salt": "yoursecretsalf",
"mysql_password": "...",
// ....
}
Then you can define a staging deploy job as such in .gitlab-ci.yml::
image: yourlabs/python
# example running tasks/update.yml, using the repo as role
script: bigsudo . update $STAGING_HOST $STAGING_VARS
# example running playbook update.yml
script: bigsudo ./update.yml $STAGING_HOST $STAGING_VARS
"""Apply a role or playbook without inventory."""
import cli2
import json
from pathlib import Path
import re
import requests
import shlex
import subprocess
import sys
import os
import yaml
M = '((?P<user>[^@]*)@)?(?P<host>([a-z0-9_-]+\.[^/]*)+)?/?(?P<path>[^/]+/.*)' # noqa
def _argv(*hosts, **variables):
"""Return generated ansible args."""
if ('v' in console_script.parser.dashargs
or 'vv' in console_script.parser.dashargs):
for dasharg in console_script.parser.dashargs:
if re.match('^v+$', dasharg):
verbose = True
if verbose:
os.environ['ANSIBLE_STDOUT_CALLBACK'] = 'debug'
argv = ['ansible-playbook'] + [
......@@ -21,28 +31,131 @@ def _argv(*hosts, **variables):
if arg.startswith('-')
]
if not hosts:
hosts = ('localhost',)
# enforce py3 here ?
# argv += ['-e', 'ansible_python_interpreter=python3']
if hosts == ('localhost',):
hosts = hosts or ('localhost',)
inv = []
user = None
for host in hosts:
if '@' in host:
parts = host.split('@')
if parts[0]:
user = parts[0]
inv.append(parts[1])
else:
inv.append(host)
if inv == ['localhost']:
argv += ['-c', 'local']
if hosts:
argv += ['-i', ','.join(hosts) + ',']
if user:
argv += ['-u', user]
if inv:
argv += ['-i', ','.join(inv) + ',']
for key, value in variables.items():
if not isinstance(value, str):
value = json.dumps(value)
argv += ['-e', key + '=' + shlex.quote(value)]
return argv
def roleinstall(role):
"""Install a role with ansible-galaxy"""
rolespath = os.path.join(os.getenv('HOME'), '.ansible', 'roles')
if not os.path.exists(rolespath):
os.makedirs(rolespath)
if getattr(roleinstall, '_cache', None) is None:
# prevent galaxy from crashing if role already installed
out = subprocess.check_output('ansible-galaxy list', shell=True)
roleinstall._cache = dict()
for roleout in out.decode('utf8').split('\n'):
ma = re.match('- (?P<name>[^,]*), (?P<version>.*)', roleout)
if not ma:
continue
roleinstall._cache[ma.group('name')] = ma.group('version')
rolename = role.rstrip('/').split('/')[-1]
if rolename in roleinstall._cache:
return
gitmatch = re.match(M, role)
def galaxyinstall(*args):
print(
subprocess.check_output(
'ansible-galaxy install ' + ' '.join(args),
shell=True
).decode('utf8')
)
if os.path.exists(role):
rolepath = Path(role)
metapath = rolepath / 'meta' / 'main.yml'
if os.path.exists(metapath):
with open(metapath, 'r') as f:
metadata = yaml.safe_load(f.read())
if 'role_name' in metadata.get('galaxy_info', {}):
rolename = metadata['galaxy_info']['role_name']
target = Path(os.getenv('HOME')) / '.ansible' / 'roles' / rolename
if target.exists():
print(f'{target} already in place, not overwriting')
else:
print(f'{target} -> {rolepath.resolve()}')
os.symlink(rolepath.resolve(), target)
elif '/' in role:
if not gitmatch:
host = 'github.com'
user = 'git'
path = rolename
else:
host = gitmatch.group('host')
user = gitmatch.group('user') or 'git'
path = gitmatch.group('path')
# try an ssh clone, fallback to https which will work if public
try:
galaxyinstall(f'git+ssh://{user}@{host}/{path}')
except subprocess.CalledProcessError:
galaxyinstall(f'git+https://{host}/{path}')
else:
galaxyinstall(role)
reqpath = os.path.join(
os.getenv('HOME'),
'.ansible',
'roles',
rolename,
'requirements.yml',
)
if os.path.exists(reqpath):
galaxyinstall('-r', reqpath)
with open(reqpath, 'r') as f:
data = yaml.safe_load(f)
for dependency in data.get('softdependencies', []):
roleinstall(dependency)
def role(role, *hosts, **variables):
"""
Apply a role.
This will use the bundled generic role application playbook.
"""
regexp = '((?P<scheme>[^+]*)\+)?(?P<url>https?([^,]*))(,(?P<ref>.*))?$'
regexp = '((?P<scheme>[^+]*)\+)?(?P<url>https?([^,]*))(,(?P<ref>.*))?$' # noqa
match = re.match(regexp, role)
if match:
parts = match.group('url').split('/')
......@@ -52,19 +165,18 @@ def role(role, *hosts, **variables):
name = role
if not os.path.exists(role):
out = subprocess.check_output([
'ansible-galaxy', 'list', name
])
if b'not found' in out:
print(subprocess.check_output([
'ansible-galaxy', 'install', role
]))
roleinstall(role)
elif name.startswith('./') or name == '.':
name = os.path.join(os.getcwd(), role[1:])
taskfile = console_script.parser.options.get('taskfile', 'main')
argv = _argv(*hosts, **variables)
argv += ['-e', 'apply_role=' + name]
if os.path.exists(name):
rolename = name
else:
rolename = name.split('/')[-1].split(',')[0]
argv += ['-e', 'apply_role=' + rolename]
playbook = os.path.join(os.path.dirname(__file__), 'role-all.yml')
argv.append(playbook)
print(' '.join(argv))
......@@ -94,7 +206,7 @@ def tasks(tasks, *hosts, **variables):
tasks = os.path.abspath(tasks)
argv = _argv(*hosts, **variables)
argv += ['-e', 'apply_tasks=' + tasks]
argv += ['-e', 'apply_tasks=' + (tasks or ['main'])]
playbook = os.path.join(os.path.dirname(__file__), 'tasks.yml')
argv.append(playbook)
print(' '.join(argv))
......@@ -120,6 +232,7 @@ def playbook(playbook, *hosts, **variables):
argv = _argv(*hosts, **variables)
argv.append(playbook)
print(' '.join(argv))
os.environ['ANSIBLE_STDOUT_CALLBACK'] = 'debug'
p = subprocess.Popen(
argv,
stderr=sys.stderr,
......@@ -129,6 +242,43 @@ def playbook(playbook, *hosts, **variables):
p.communicate()
return p.returncode
def run(source, *hosts_or_tasks, **variables):
"""
This commands executes a role's tasks with variables from the CLI.
# this will execute repo/tasks/main.yml
bigsudo github.com/your/repo @localhost somearg=foo
# this will execute ref yourbranch's repo/tasks/update.yml
bigsudo github.com/your/repo,yourbranch @localhost somearg=foo update
This command requires that the repository contains a meta/main.yml with:
galaxy_info:
author: yourname
description: yourdescription
Note that you can generate one with the ansible-galaxy init command.
"""
hosts = []
tasks = []
for arg in hosts_or_tasks:
if '@' in arg:
hosts.append(arg)
else:
tasks.append(arg)
kwargs = dict(apply_tasks=tasks or ['main'])
kwargs.update(variables)
if source.endswith('.yml'):
return playbook(source, *hosts, **kwargs)
else:
return role(source, *hosts, **kwargs)
console_script = cli2.ConsoleScript(
__doc__,
).add_module('ansible_apply.console_script')
default_command='run',
).add_module('bigsudo.console_script')
......@@ -3,7 +3,11 @@
- include: bootstrap.yml
- hosts: '*'
vars:
apply_tasks: '{{ apply_tasks|default(["main"]) }}'
tasks:
- debug: msg='Applying {{ apply_role }} tasks {{ apply_tasks }}'
- include_role:
name: '{{ apply_role }}'
tasks_from: '{{ apply_tasks|default("main") }}.yml'
tasks_from: '{{ item }}.yml'
loop: '{{ apply_tasks }}'
[console_scripts]
ansible-apply = ansible_apply.console_script:console_script
bigsudo = bigsudo.console_script:console_script
#!/usr/bin/env bigsudo
---
- hosts: '*'
tasks:
- shell: echo hello > {{ lookup('env', 'PWD') }}/playbook.out
- debug: msg='Success running example playbook !'
from setuptools import setup
setup(
name='ansible-apply',
name='bigsudo',
versioning='post',
url='https://yourlabs.io/oss/ansible-apply',
url='https://yourlabs.io/oss/bigsudo',
setup_requires='setupmeta',
keywords='automation cli',
keywords='automation cli ansible',
python_requires='>=3',
include_package_data=True,
)
command: bigsudo yourlabs.io/oss/yourlabs.bigsudo-example @localhost
retcode: 0
stdout:
[WARNING]: - the configured path /usr/share/ansible/roles does not exist.
[WARNING]: - the configured path /etc/ansible/roles does not exist.
ansible-playbook -c local -i localhost, -e apply_role=yourlabs.bigsudo-example /home/jpic/bigsudo/bigsudo/role-all.yml
[DEPRECATION WARNING]: 'include' for playbook includes. You should use
'import_playbook' instead. This feature will be removed in version 2.8.
Deprecation warnings can be disabled by setting deprecation_warnings=False in
ansible.cfg.
PLAY [*] ***********************************************************************
TASK [Install python for ansible to work] **************************************
changed: [localhost]
PLAY [*] ***********************************************************************
TASK [Gathering Facts] *********************************************************
ok: [localhost]
TASK [include_role : {{ apply_role }}] *****************************************
TASK [yourlabs.bigsudo-example : Write a ~/bigsudo.main file] ******************
changed: [localhost]
PLAY RECAP *********************************************************************
localhost : ok=3 changed=2 unreachable=0 failed=0
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment