14 Commits

Author SHA1 Message Date
towfiqi
c3ddb9d3c3 chore(release): 1.0.0 2024-02-09 21:47:39 +06:00
towfiqi
dbf540cfdb fix: Resolves missing Keyword Loading Spinner issue. 2024-02-09 21:18:23 +06:00
towfiqi
1f0831ed13 chore: Updates vulnerable dependencies. 2024-02-09 00:56:59 +06:00
towfiqi
b4ad69baaa feat: Adds Serper.dev integration
closes #138
2024-02-09 00:43:28 +06:00
towfiqi
f04b10cf6b feat: Adds the ability to setup Search Console through the UI.
- Adds the ability to add domain specific Search Console API Info through the Domain Settings panel.
- Adds the ability to add global Search Console API Info through the App Settings Panel.
- Adds better Search Console Error logging.
- Changes the App Settings Sidebar UI.
- Changers the Domain Settings Modal UI.
- Replaces html Input field with custom InputField component.
- Adds a new /domain api route to get the full domain info which includes the domain level Search console API.

closes #59, #146
2024-02-08 22:14:24 +06:00
towfiqi
b2e97b2ebe feat: Adds the Ability to set Search Console Property type via Domain Settings.
- Previously only domain properties worked with SerpBear. This feature adds the ability to add URL properties as well.
- Adds a new field "search_console" in Domain Table.
- Adds a new Search Console option in Domain Settings Modal UI.
- When the  new "This is a URL Property" option is enabled, the exact Property URL should be provided.

closes #50
2024-02-06 23:42:28 +06:00
towfiqi
1041cb3c0b feat: Adds ValueSerp Integration.
closes #105, #106
2024-02-06 13:32:24 +06:00
towfiqi
3719f21d98 feat: Adds the ability for city level scraping for scapers that allow it.
- Only available for scrapers that allows custom location or city level scraping.
- When a city level keyword is added the city name is displayed in the keyword title.

closes #139, #151
2024-02-06 13:22:32 +06:00
towfiqi
444ba5d461 refactor: Adds Keyword Table migration to add new fields.
- Adds city, latlong and settings fields to Keyword table.
2024-02-04 23:43:13 +06:00
towfiqi
dd54e535c9 build: adds database migration method. 2024-02-04 23:39:39 +06:00
towfiqi
34d121dac7 refactor: removes unnecessary useEffect 2024-02-04 10:27:44 +06:00
towfiqi
3c2a1b8a5b feat: adds the ability to add url as a domain.
You can now track specific marketplace/social domain URLs. For example a reddit.com post, an amazon.com product, github repo etc.

closes: #53, #90, #119
2024-02-03 20:17:51 +06:00
towfiqi
e2ecdef10e Chore: Fixes Typo 2024-02-03 10:15:07 +06:00
towfiqi
633ab2c467 fix: Resolves Keywords filter crashing issue. 2024-02-03 10:14:46 +06:00
54 changed files with 1717 additions and 300 deletions

View File

@@ -13,6 +13,7 @@
"arrow-body-style":"off",
"max-len": ["error", {"code": 150, "ignoreComments": true, "ignoreUrls": true}],
"import/no-extraneous-dependencies": "off",
"no-unused-vars": "off",
"import/extensions": [
"error",
"ignorePackages",

8
.sequelizerc Normal file
View File

@@ -0,0 +1,8 @@
const path = require('path');
module.exports = {
'config': path.resolve('database', 'config.js'),
'models-path': path.resolve('database', 'models'),
'seeders-path': path.resolve('database', 'seeders'),
'migrations-path': path.resolve('database', 'migrations')
};

View File

@@ -2,6 +2,24 @@
All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
## [1.0.0](https://github.com/towfiqi/serpbear/compare/v0.3.4...v1.0.0) (2024-02-09)
### Features
* Adds Serper.dev integration ([b4ad69b](https://github.com/towfiqi/serpbear/commit/b4ad69baaa0f865938f8b0eace6732a9e6b1b381)), closes [#138](https://github.com/towfiqi/serpbear/issues/138)
* Adds the ability for city level scraping for scapers that allow it. ([3719f21](https://github.com/towfiqi/serpbear/commit/3719f21d98d173219cef5656579fa0e5340ccdbf)), closes [#139](https://github.com/towfiqi/serpbear/issues/139) [#151](https://github.com/towfiqi/serpbear/issues/151)
* adds the ability to add url as a domain. ([3c2a1b8](https://github.com/towfiqi/serpbear/commit/3c2a1b8a5b8a2a4a2179a5031582f8202c2e494a)), closes [#53](https://github.com/towfiqi/serpbear/issues/53) [#90](https://github.com/towfiqi/serpbear/issues/90) [#119](https://github.com/towfiqi/serpbear/issues/119)
* Adds the Ability to set Search Console Property type via Domain Settings. ([b2e97b2](https://github.com/towfiqi/serpbear/commit/b2e97b2ebec380f0edf7ddc0640c2126eff006ac)), closes [#50](https://github.com/towfiqi/serpbear/issues/50)
* Adds the ability to setup Search Console through the UI. ([f04b10c](https://github.com/towfiqi/serpbear/commit/f04b10cf6b065e3023965112a60e0aa702212a4b)), closes [#59](https://github.com/towfiqi/serpbear/issues/59) [#146](https://github.com/towfiqi/serpbear/issues/146)
* Adds ValueSerp Integration. ([1041cb3](https://github.com/towfiqi/serpbear/commit/1041cb3c0bb69e9034696624e03433be28e83ac6)), closes [#105](https://github.com/towfiqi/serpbear/issues/105) [#106](https://github.com/towfiqi/serpbear/issues/106)
### Bug Fixes
* Resolves Keywords filter crashing issue. ([633ab2c](https://github.com/towfiqi/serpbear/commit/633ab2c467be5b7b86d4547ae0c59034e595a42d))
* Resolves missing Keyword Loading Spinner issue. ([dbf540c](https://github.com/towfiqi/serpbear/commit/dbf540cfdb16ddb02c9d26618e3680d34799f57f))
### [0.3.4](https://github.com/towfiqi/serpbear/compare/v0.3.3...v0.3.4) (2024-01-15)

View File

@@ -30,13 +30,17 @@ COPY --from=builder --chown=nextjs:nodejs /app/.next/static ./.next/static
# setup the cron
COPY --from=builder --chown=nextjs:nodejs /app/cron.js ./
COPY --from=builder --chown=nextjs:nodejs /app/email ./email
COPY --from=builder --chown=nextjs:nodejs /app/database ./database
COPY --from=builder --chown=nextjs:nodejs /app/.sequelizerc ./.sequelizerc
COPY --from=builder --chown=nextjs:nodejs /app/entrypoint.sh ./entrypoint.sh
RUN rm package.json
RUN npm init -y
RUN npm i cryptr dotenv croner @googleapis/searchconsole
RUN npm i cryptr dotenv croner @googleapis/searchconsole sequelize-cli
RUN npm i -g concurrently
USER nextjs
EXPOSE 3000
ENTRYPOINT ["/app/entrypoint.sh"]
CMD ["concurrently","node server.js", "node cron.js"]

View File

@@ -42,6 +42,7 @@ The App uses third party website scrapers like ScrapingAnt, ScrapingRobot, Searc
| serpapi.com | From $50/mo** | From 5,000/mo** | Yes |
| spaceserp.com | $59/lifetime | 15,000/mo | Yes |
| SearchApi.io | From $40/mo | From 10,000/mo | Yes |
| valueserp.com | Pay As You Go | $2.50/1000 req | No |
(*) Free upto a limit. If you are using ScrapingAnt you can lookup 10,000 times per month for free.
(**) Free up to 100 per month. Paid from 5,000 to 10,000,000+ per month.

View File

@@ -253,6 +253,27 @@ const Icon = ({ type, color = 'currentColor', size = 16, title = '', classes = '
<path d="M15.75 9h3v2.25h-3z" fill={color} />
</svg>
}
{type === 'email'
&& <svg {...xmlnsProps} width={size} viewBox="0 0 24 24">
<path fill={color} d="M22 6c0-1.1-.9-2-2-2H4c-1.1 0-2 .9-2 2v12c0 1.1.9 2 2 2h16c1.1 0 2-.9 2-2zm-2 0l-8 5l-8-5zm0 12H4V8l8 5l8-5z" />
</svg>
}
{type === 'scraper'
&& <svg {...xmlnsProps} width={size} viewBox="0 0 16 16">
<path fill={color} d="M1 3.5A2.5 2.5 0 0 1 3.5 1h7A2.5 2.5 0 0 1 13 3.5v1.53a4.538 4.538 0 0 0-1-.004V5H2v5.5A1.5 1.5 0 0 0 3.5 12h2.954l-.72.72a2.52 2.52 0 0 0-.242.28H3.5A2.5 2.5 0 0 1 1 10.5zm7.931 3.224l-.577-.578a.5.5 0 1 0-.708.708l.745.744c.144-.306.324-.6.54-.874M2 4h10v-.5A1.5 1.5 0 0 0 10.5 2h-7A1.5 1.5 0 0 0 2 3.5zm4.354 2.854a.5.5 0 1 0-.708-.708l-2 2a.5.5 0 0 0 0 .708l2 2a.5.5 0 0 0 .708-.708L4.707 8.5zm6.538-.83c.366.042.471.48.21.742l-.975.975a1.507 1.507 0 1 0 2.132 2.132l.975-.975c.261-.261.7-.156.742.21a3.518 3.518 0 0 1-4.676 3.723l-2.726 2.727a1.507 1.507 0 1 1-2.132-2.132L9.168 10.7a3.518 3.518 0 0 1 3.724-4.676" />
</svg>
}
{type === 'city'
&& <svg {...xmlnsProps} width={size} viewBox="0 0 48 48">
<g fill="none">
<path stroke={color} strokeLinecap="round" strokeLinejoin="round" strokeWidth={4} d="M4 42h40"></path>
<rect width={8} height={16} x={8} y={26} stroke={color} strokeLinejoin="round" strokeWidth={4} rx={2}></rect>
<path stroke={color} strokeLinecap="square" strokeLinejoin="round" strokeWidth={4} d="M12 34h1"></path>
<rect width={24} height={38} x={16} y={4} stroke={color} strokeLinejoin="round" strokeWidth={4} rx={2}></rect>
<path fill={color} d="M22 10h4v4h-4zm8 0h4v4h-4zm-8 7h4v4h-4zm8 0h4v4h-4zm0 7h4v4h-4zm0 7h4v4h-4z"></path>
</g>
</svg>
}
</span>
);
};

View File

@@ -0,0 +1,28 @@
type InputFieldProps = {
label: string;
value: string;
onChange: Function;
placeholder?: string;
classNames?: string;
hasError?: boolean;
}
const InputField = ({ label = '', value = '', placeholder = '', onChange, hasError = false }: InputFieldProps) => {
const labelStyle = 'mb-2 font-semibold inline-block text-sm text-gray-700 capitalize';
return (
<div className="field--input w-full relative flex justify-between items-center">
<label className={labelStyle}>{label}</label>
<input
className={`p-2 border border-gray-200 rounded focus:outline-none w-[210px]
focus:border-blue-200 ${hasError ? ' border-red-400 focus:border-red-400' : ''} `}
type={'text'}
value={value}
onChange={(event) => onChange(event.target.value)}
autoComplete="off"
placeholder={placeholder}
/>
</div>
);
};
export default InputField;

View File

@@ -6,10 +6,11 @@ type ModalProps = {
children: React.ReactNode,
width?: string,
title?: string,
verticalCenter?: boolean,
closeModal: Function,
}
const Modal = ({ children, width = '1/2', closeModal, title }:ModalProps) => {
const Modal = ({ children, width = '1/2', closeModal, title, verticalCenter = false }:ModalProps) => {
useOnKey('Escape', closeModal);
const closeOnBGClick = (e:React.SyntheticEvent) => {
@@ -21,8 +22,9 @@ const Modal = ({ children, width = '1/2', closeModal, title }:ModalProps) => {
return (
<div className='modal fixed top-0 left-0 bg-white/[.7] w-full h-screen z-50' onClick={closeOnBGClick}>
<div
className={`modal__content max-w-[340px] absolute top-1/4 left-0 right-0 ml-auto mr-auto w-${width}
lg:max-w-md bg-white shadow-md rounded-md p-5 border-t-[1px] border-gray-100 text-base`}>
className={`modal__content max-w-[340px] absolute left-0 right-0 ml-auto mr-auto w-${width}
lg:max-w-md bg-white shadow-md rounded-md p-5 border-t-[1px] border-gray-100 text-base
${verticalCenter ? ' top-1/2 translate-y-[-50%]' : 'top-1/4'}`}>
{title && <h3 className=' font-semibold mb-3'>{title}</h3>}
<button
className='modal-close absolute right-2 top-2 p-2 cursor-pointer text-gray-400 transition-all

View File

@@ -14,15 +14,15 @@ const SecretField = ({ label = '', value = '', placeholder = '', onChange, hasEr
const [showValue, setShowValue] = useState(false);
const labelStyle = 'mb-2 font-semibold inline-block text-sm text-gray-700 capitalize';
return (
<div className="settings__section__secret mb-5 relative">
<div className="settings__section__secret mb-5 relative flex justify-between items-center">
<label className={labelStyle}>{label}</label>
<span
className="absolute top-8 right-0 px-2 py-1 cursor-pointer text-gray-400 select-none"
className="absolute top-1 right-0 px-2 py-1 cursor-pointer text-gray-400 select-none"
onClick={() => setShowValue(!showValue)}>
<Icon type={showValue ? 'eye-closed' : 'eye'} size={18} />
</span>
<input
className={`w-full p-2 border border-gray-200 rounded mb-3 focus:outline-none
className={`w-[210px] p-2 border border-gray-200 rounded focus:outline-none
focus:border-blue-200 ${hasError ? ' border-red-400 focus:border-red-400' : ''} `}
type={showValue ? 'text' : 'password'}
value={value}

View File

@@ -9,6 +9,7 @@ type SelectFieldProps = {
defaultLabel: string,
options: SelectionOption[],
selected: string[],
label?: string,
multiple?: boolean,
updateField: Function,
minWidth?: number,
@@ -28,6 +29,7 @@ const SelectField = (props: SelectFieldProps) => {
maxHeight = 96,
rounded = 'rounded-3xl',
flags = false,
label = '',
emptyMsg = '' } = props;
const [showOptions, setShowOptions] = useState<boolean>(false);
@@ -66,12 +68,13 @@ const SelectField = (props: SelectFieldProps) => {
};
return (
<div className="select font-semibold text-gray-500">
<div className="select font-semibold text-gray-500 relative flex justify-between items-center">
{label && <label className='mb-2 font-semibold inline-block text-sm text-gray-700 capitalize'>{label}</label>}
<div
className={`selected flex border ${rounded} p-1.5 px-4 cursor-pointer select-none w-[180px] min-w-[${minWidth}px]
className={`selected flex border ${rounded} p-1.5 px-4 cursor-pointer select-none w-[210px] min-w-[${minWidth}px]
${showOptions ? 'border-indigo-200' : ''}`}
onClick={() => setShowOptions(!showOptions)}>
<span className={`w-[${minWidth - 30}px] inline-block truncate mr-2 capitalize`}>
<span className={'w-full inline-block truncate mr-2 capitalize'}>
{selected.length > 0 ? (selectedLabels.slice(0, 2).join(', ')) : defaultLabel}
</span>
{multiple && selected.length > 2
@@ -80,7 +83,7 @@ const SelectField = (props: SelectFieldProps) => {
</div>
{showOptions && (
<div
className={`select_list mt-1 border absolute min-w-[${minWidth}px]
className={`select_list mt-1 border absolute min-w-[${minWidth}px] top-[30px] right-0 w-[210px]
${rounded === 'rounded-3xl' ? 'rounded-lg' : rounded} max-h-${maxHeight} bg-white z-50 overflow-y-auto styled-scrollbar`}>
{options.length > 20 && (
<div className=''>

View File

@@ -0,0 +1,32 @@
type ToggleFieldProps = {
label: string;
value: string;
onChange: (bool:boolean) => void ;
classNames?: string;
}
const ToggleField = ({ label = '', value = '', onChange, classNames = '' }: ToggleFieldProps) => {
return (
<div className={`field--toggle w-full relative ${classNames}`}>
<label className="relative inline-flex items-center cursor-pointer w-full justify-between">
<span className="text-sm font-medium text-gray-900 dark:text-gray-300 w-56">{label}</span>
<input
type="checkbox"
value={value}
checked={!!value}
className="sr-only peer"
onChange={() => onChange(!value)}
/>
<div className="relative rounded-3xl w-9 h-5 bg-gray-200 peer-focus:outline-none peer-focus:ring-4
peer-focus:ring-blue-300 dark:peer-focus:ring-blue-800rounded-full peer dark:bg-gray-700
peer-checked:after:translate-x-full peer-checked:after:border-white after:content-['']
after:absolute after:top-[2px] after:left-[2px] after:bg-white after:border-gray-300
after:border after:rounded-full after:h-4 after:w-4
after:transition-all dark:border-gray-600 peer-checked:bg-blue-600"></div>
</label>
</div>
);
};
export default ToggleField;

View File

@@ -1,7 +1,7 @@
import React, { useState } from 'react';
import Modal from '../common/Modal';
import { useAddDomain } from '../../services/domains';
import { isValidDomain } from '../../utils/client/validators';
import { isValidUrl } from '../../utils/client/validators';
type AddDomainProps = {
domains: DomainType[],
@@ -16,24 +16,30 @@ const AddDomain = ({ closeModal, domains = [] }: AddDomainProps) => {
const addDomain = () => {
setNewDomainError('');
const existingDomains = domains.map((d) => d.domain);
const insertedDomains = newDomain.split('\n');
const insertedURLs = newDomain.split('\n');
const domainsTobeAdded:string[] = [];
const invalidDomains:string[] = [];
insertedDomains.forEach((dom) => {
const domain = dom.trim();
if (isValidDomain(domain)) {
if (!existingDomains.includes(domain)) {
domainsTobeAdded.push(domain);
insertedURLs.forEach((url) => {
const theURL = url.trim();
if (isValidUrl(theURL)) {
const domURL = new URL(theURL);
const isDomain = domURL.pathname === '/';
if (isDomain && !existingDomains.includes(domURL.host)) {
domainsTobeAdded.push(domURL.host);
}
if (!isDomain && !existingDomains.includes(domURL.href)) {
const cleanedURL = domURL.href.replace('https://', '').replace('http://', '').replace(/^\/+|\/+$/g, '');
domainsTobeAdded.push(cleanedURL);
}
} else {
invalidDomains.push(domain);
invalidDomains.push(theURL);
}
});
if (invalidDomains.length > 0) {
setNewDomainError(`Please Insert Valid Domain names. Invalid Domains: ${invalidDomains.join(', ')}`);
setNewDomainError(`Please Insert Valid Domain URL. Invalid URLs: ${invalidDomains.join(', ')}`);
} else if (domainsTobeAdded.length > 0) {
// TODO: Domain Action
addMutate(domainsTobeAdded);
console.log('domainsTobeAdded :', domainsTobeAdded);
addMutate(domainsTobeAdded);
}
};
@@ -45,11 +51,11 @@ const AddDomain = ({ closeModal, domains = [] }: AddDomainProps) => {
return (
<Modal closeModal={() => { closeModal(false); }} title={'Add New Domain'}>
<div data-testid="adddomain_modal">
<h4 className='text-sm mt-4'>Domain Names</h4>
<h4 className='text-sm mt-4'>Domain URL</h4>
<textarea
className={`w-full h-40 border rounded border-gray-200 p-4 outline-none
focus:border-indigo-300 ${newDomainError ? ' border-red-400 focus:border-red-400' : ''}`}
placeholder="Type or Paste Domains here. Insert Each Domain in a New line."
placeholder="Type or Paste URLs here. Insert Each URL in a New line."
value={newDomain}
autoFocus={true}
onChange={handleDomainInput}>

View File

@@ -38,7 +38,7 @@ const DomainItem = ({ domain, selected, isConsoleIntegrated = false, thumb, upda
/>
</div>
<div className="domain_details flex-1">
<h3 className='font-semibold text-base mb-2'>{domain.domain}</h3>
<h3 className='font-semibold text-base mb-2 max-w-[200px] text-ellipsis overflow-hidden' title={domain.domain}>{domain.domain}</h3>
{keywordsUpdated && (
<span className=' text-gray-600 text-xs'>
Updated <TimeAgo title={dayjs(keywordsUpdated).format('DD-MMM-YYYY, hh:mm:ss A')} date={keywordsUpdated} />

View File

@@ -2,7 +2,9 @@ import { useRouter } from 'next/router';
import { useState } from 'react';
import Icon from '../common/Icon';
import Modal from '../common/Modal';
import { useDeleteDomain, useUpdateDomain } from '../../services/domains';
import { useDeleteDomain, useFetchDomain, useUpdateDomain } from '../../services/domains';
import InputField from '../common/InputField';
import SelectField from '../common/SelectField';
type DomainSettingsProps = {
domain:DomainType|false,
@@ -16,25 +18,27 @@ type DomainSettingsError = {
const DomainSettings = ({ domain, closeModal }: DomainSettingsProps) => {
const router = useRouter();
const [currentTab, setCurrentTab] = useState<'notification'|'searchconsole'>('notification');
const [showRemoveDomain, setShowRemoveDomain] = useState<boolean>(false);
const [settingsError, setSettingsError] = useState<DomainSettingsError>({ type: '', msg: '' });
const [domainSettings, setDomainSettings] = useState<DomainSettings>(() => ({
notification_interval: domain && domain.notification_interval ? domain.notification_interval : 'never',
notification_emails: domain && domain.notification_emails ? domain.notification_emails : '',
search_console: domain && domain.search_console ? JSON.parse(domain.search_console) : {
property_type: 'domain', url: '', client_email: '', private_key: '',
},
}));
const { mutate: updateMutate } = useUpdateDomain(() => closeModal(false));
const { mutate: deleteMutate } = useDeleteDomain(() => {
closeModal(false);
router.push('/domains');
const { mutate: updateMutate, error: domainUpdateError, isLoading: isUpdating } = useUpdateDomain(() => closeModal(false));
const { mutate: deleteMutate } = useDeleteDomain(() => { closeModal(false); router.push('/domains'); });
// Get the Full Domain Data along with the Search Console API Data.
useFetchDomain(router, domain && domain.domain ? domain.domain : '', (domainObj:DomainType) => {
const currentSearchConsoleSettings = domainObj.search_console && JSON.parse(domainObj.search_console);
setDomainSettings({ ...domainSettings, search_console: currentSearchConsoleSettings || domainSettings.search_console });
});
const updateNotiEmails = (event:React.FormEvent<HTMLInputElement>) => {
setDomainSettings({ ...domainSettings, notification_emails: event.currentTarget.value });
};
const updateDomain = () => {
console.log('Domain: ');
let error: DomainSettingsError | null = null;
if (domainSettings.notification_emails) {
const notification_emails = domainSettings.notification_emails.split(',');
@@ -55,24 +59,103 @@ const DomainSettings = ({ domain, closeModal }: DomainSettingsProps) => {
}
};
const tabStyle = `inline-block px-4 py-2 rounded-md mr-3 cursor-pointer text-sm select-none z-10
text-gray-600 border border-b-0 relative top-[1px] rounded-b-none`;
return (
<div>
<Modal closeModal={() => closeModal(false)} title={'Domain Settings'} width="[500px]">
<Modal closeModal={() => closeModal(false)} title={'Domain Settings'} width="[500px]" verticalCenter={currentTab === 'searchconsole'} >
<div data-testid="domain_settings" className=" text-sm">
<div className="mb-6 flex justify-between items-center">
<h4>Notification Emails
{settingsError.type === 'email' && <span className="text-red-500 font-semibold ml-2">{settingsError.msg}</span>}
</h4>
<input
className={`border w-46 text-sm transition-all rounded p-1.5 px-4 outline-none ring-0
${settingsError.type === 'email' ? ' border-red-300' : ''}`}
type="text"
placeholder='Your Emails'
onChange={updateNotiEmails}
value={domainSettings.notification_emails || ''}
/>
<div className=' mt-3 mb-5 border border-slate-200 px-2 py-4 pb-0
relative left-[-20px] w-[calc(100%+40px)] border-l-0 border-r-0 bg-[#f8f9ff]'>
<ul>
<li
className={`${tabStyle} ${currentTab === 'notification' ? ' bg-white text-blue-600 border-slate-200' : 'border-transparent'} `}
onClick={() => setCurrentTab('notification')}>
<Icon type='email' /> Notification
</li>
<li
className={`${tabStyle} ${currentTab === 'searchconsole' ? ' bg-white text-blue-600 border-slate-200' : 'border-transparent'}`}
onClick={() => setCurrentTab('searchconsole')}>
<Icon type='google' /> Search Console
</li>
</ul>
</div>
<div>
{currentTab === 'notification' && (
<div className="mb-4 flex justify-between items-center w-full">
<InputField
label='Notification Emails'
onChange={(emails:string) => setDomainSettings({ ...domainSettings, notification_emails: emails })}
value={domainSettings.notification_emails || ''}
placeholder='Your Emails'
/>
</div>
)}
{currentTab === 'searchconsole' && (
<>
<div className="mb-4 flex justify-between items-center w-full">
<label className='mb-2 font-semibold inline-block text-sm text-gray-700 capitalize'>Property Type</label>
<SelectField
options={[{ label: 'Domain', value: 'domain' }, { label: 'URL', value: 'url' }]}
selected={[domainSettings.search_console?.property_type || 'domain']}
defaultLabel="Select Search Console Property Type"
updateField={(updated:['domain'|'url']) => setDomainSettings({
...domainSettings,
search_console: { ...(domainSettings.search_console as DomainSearchConsole), property_type: updated[0] || 'domain' },
})}
multiple={false}
rounded={'rounded'}
/>
</div>
{domainSettings?.search_console?.property_type === 'url' && (
<div className="mb-4 flex justify-between items-center w-full">
<InputField
label='Property URL (Required)'
onChange={(url:string) => setDomainSettings({
...domainSettings,
search_console: { ...(domainSettings.search_console as DomainSearchConsole), url },
})}
value={domainSettings?.search_console?.url || ''}
placeholder='Search Console Property URL. eg: https://mywebsite.com/'
/>
</div>
)}
<div className="mb-4 flex justify-between items-center w-full">
<InputField
label='Search Console Client Email'
onChange={(client_email:string) => setDomainSettings({
...domainSettings,
search_console: { ...(domainSettings.search_console as DomainSearchConsole), client_email },
})}
value={domainSettings?.search_console?.client_email || ''}
placeholder='myapp@appspot.gserviceaccount.com'
/>
</div>
<div className="mb-4 flex flex-col justify-between items-center w-full">
<label className='mb-2 font-semibold block text-sm text-gray-700 capitalize w-full'>Search Console Private Key</label>
<textarea
className={`w-full p-2 border border-gray-200 rounded mb-3 text-xs
focus:outline-none h-[100px] focus:border-blue-200`}
value={domainSettings?.search_console?.private_key || ''}
placeholder={'-----BEGIN PRIVATE KEY-----/ssssaswdkihad....'}
onChange={(event) => setDomainSettings({
...domainSettings,
search_console: { ...(domainSettings.search_console as DomainSearchConsole), private_key: event.target.value },
})}
/>
</div>
</>
)}
</div>
{!isUpdating && (domainUpdateError as Error)?.message && (
<div className='w-full mt-4 p-3 text-sm bg-red-50 text-red-700'>{(domainUpdateError as Error).message}</div>
)}
{!isUpdating && settingsError?.msg && (
<div className='w-full mt-4 p-3 text-sm bg-red-50 text-red-700'>{settingsError.msg}</div>
)}
</div>
<div className="flex justify-between border-t-[1px] border-gray-100 mt-8 pt-4 pb-0">
<button
className="text-sm font-semibold text-red-500"
@@ -80,9 +163,9 @@ const DomainSettings = ({ domain, closeModal }: DomainSettingsProps) => {
<Icon type="trash" /> Remove Domain
</button>
<button
className='text-sm font-semibold py-2 px-5 rounded cursor-pointer bg-blue-700 text-white'
onClick={() => updateDomain()}>
Update Settings
className={`text-sm font-semibold py-2 px-5 rounded cursor-pointer bg-blue-700 text-white ${isUpdating ? 'cursor-not-allowed' : ''}`}
onClick={() => !isUpdating && updateDomain()}>
{isUpdating && <Icon type='loading' />} Update Settings
</button>
</div>
</Modal>

View File

@@ -117,7 +117,7 @@ const SCInsight = ({ insight, isLoading = true, isConsoleIntegrated = true, doma
)}
{!isConsoleIntegrated && (
<p className=' p-9 pt-[10%] text-center text-gray-500'>
Google Search has not been Integrated yet. Please follow <a className='text-indigo-600 underline' href='https://docs.serpbear.com/miscellaneous/integrate-google-search-console' target="_blank" rel='noreferrer'>These Steps</a> to integrate Google Search Data for this Domain.
Google Search Console has not been Integrated yet. Please follow <a className='text-indigo-600 underline' href='https://docs.serpbear.com/miscellaneous/integrate-google-search-console' target="_blank" rel='noreferrer'>These Steps</a> to integrate Google Search Data for this Domain.
</p>
)}
</div>

View File

@@ -7,6 +7,8 @@ import { useAddKeywords } from '../../services/keywords';
type AddKeywordsProps = {
keywords: KeywordType[],
scraperName: string,
allowsCity: boolean,
closeModal: Function,
domain: string
}
@@ -17,9 +19,10 @@ type KeywordsInput = {
country: string,
domain: string,
tags: string,
city?:string,
}
const AddKeywords = ({ closeModal, domain, keywords }: AddKeywordsProps) => {
const AddKeywords = ({ closeModal, domain, keywords, scraperName = '', allowsCity = false }: AddKeywordsProps) => {
const [error, setError] = useState<string>('');
const defCountry = localStorage.getItem('default_country') || 'US';
const [newKeywordsData, setNewKeywordsData] = useState<KeywordsInput>({ keywords: '', device: 'desktop', country: defCountry, domain, tags: '' });
@@ -29,14 +32,16 @@ const AddKeywords = ({ closeModal, domain, keywords }: AddKeywordsProps) => {
const addKeywords = () => {
if (newKeywordsData.keywords) {
const keywordsArray = [...new Set(newKeywordsData.keywords.split('\n').map((item) => item.trim()).filter((item) => !!item))];
const currentKeywords = keywords.map((k) => `${k.keyword}-${k.device}-${k.country}`);
const keywordExist = keywordsArray.filter((k) => currentKeywords.includes(`${k}-${newKeywordsData.device}-${newKeywordsData.country}`));
const currentKeywords = keywords.map((k) => `${k.keyword}-${k.device}-${k.country}${k.city ? `-${k.city}` : ''}`);
const keywordExist = keywordsArray.filter((k) => currentKeywords.includes(
`${k}-${newKeywordsData.device}-${newKeywordsData.country}${newKeywordsData.city ? `-${newKeywordsData.city}` : ''}`,
));
if (keywordExist.length > 0) {
setError(`Keywords ${keywordExist.join(',')} already Exist`);
setTimeout(() => { setError(''); }, 3000);
} else {
const { device, country, domain: kDomain, tags } = newKeywordsData;
const newKeywordsArray = keywordsArray.map((nItem) => ({ keyword: nItem, device, country, domain: kDomain, tags }));
const { device, country, domain: kDomain, tags, city } = newKeywordsData;
const newKeywordsArray = keywordsArray.map((nItem) => ({ keyword: nItem, device, country, domain: kDomain, tags, city }));
addMutate(newKeywordsArray);
}
} else {
@@ -85,17 +90,28 @@ const AddKeywords = ({ closeModal, domain, keywords }: AddKeywordsProps) => {
><Icon type='mobile' /> <i className='not-italic hidden lg:inline-block'>Mobile</i></li>
</ul>
</div>
<div className='relative'>
{/* TODO: Insert Existing Tags as Suggestions */}
<input
className='w-full border rounded border-gray-200 py-2 px-4 pl-8 outline-none focus:border-indigo-300'
placeholder='Insert Tags'
placeholder='Insert Tags (Optional)'
value={newKeywordsData.tags}
onChange={(e) => setNewKeywordsData({ ...newKeywordsData, tags: e.target.value })}
/>
<span className='absolute text-gray-400 top-2 left-2'><Icon type="tags" size={16} /></span>
</div>
<div className='relative mt-2'>
<input
className={`w-full border rounded border-gray-200 py-2 px-4 pl-8
outline-none focus:border-indigo-300 ${!allowsCity ? ' cursor-not-allowed' : ''} `}
disabled={!allowsCity}
title={!allowsCity ? `Your scraper ${scraperName} doesn't have city level scraping feature.` : ''}
placeholder={`City (Optional)${!allowsCity ? `. Not avaialable for ${scraperName}.` : ''}`}
value={newKeywordsData.city}
onChange={(e) => setNewKeywordsData({ ...newKeywordsData, city: e.target.value })}
/>
<span className='absolute text-gray-400 top-2 left-2'><Icon type="city" size={16} /></span>
</div>
</div>
{error && <div className='w-full mt-4 p-3 text-sm bg-red-50 text-red-700'>{error}</div>}
<div className='mt-6 text-right text-sm font-semibold flex justify-between'>

View File

@@ -40,7 +40,7 @@ const Keyword = (props: KeywordProps) => {
scDataType = 'threeDays',
} = props;
const {
keyword, domain, ID, position, url = '', lastUpdated, country, sticky, history = {}, updating = false, lastUpdateError = false,
keyword, domain, ID, city, position, url = '', lastUpdated, country, sticky, history = {}, updating = false, lastUpdateError = false,
} = keywordData;
const [showOptions, setShowOptions] = useState(false);
const [showPositionError, setPositionError] = useState(false);
@@ -85,12 +85,12 @@ const Keyword = (props: KeywordProps) => {
return (
<div
key={keyword}
key={keyword + ID}
style={style}
className={`keyword relative py-5 px-4 text-gray-600 border-b-[1px] border-gray-200 lg:py-4 lg:px-6 lg:border-0
lg:flex lg:justify-between lg:items-center ${selected ? ' bg-indigo-50 keyword--selected' : ''} ${lastItem ? 'border-b-0' : ''}`}>
<div className=' w-3/4 lg:flex-1 lg:basis-20 lg:w-auto font-semibold cursor-pointer'>
<div className=' w-3/4 font-semibold cursor-pointer lg:flex-1 lg:basis-20 lg:w-auto lg:flex lg:items-center'>
<button
className={`p-0 mr-2 leading-[0px] inline-block rounded-sm pt-0 px-[1px] pb-[3px] border
${selected ? ' bg-blue-700 border-blue-700 text-white' : 'text-transparent'}`}
@@ -99,9 +99,10 @@ const Keyword = (props: KeywordProps) => {
<Icon type="check" size={10} />
</button>
<a
className='py-2 hover:text-blue-600'
className='py-2 hover:text-blue-600 lg:flex lg:items-center lg:w-full'
onClick={() => showKeywordDetails()}>
<span className={`fflag fflag-${country} w-[18px] h-[12px] mr-2`} title={countries[country][0]} />{keyword}
<span className={`fflag fflag-${country} w-[18px] h-[12px] mr-2`} title={countries[country][0]} />
<span className=' text-ellipsis overflow-hidden whitespace-nowrap w-[calc(100%-30px)]'>{keyword}{city ? ` (${city})` : ''}</span>
</a>
{sticky && <button className='ml-2 relative top-[2px]' title='Favorite'><Icon type="star-filled" size={16} color="#fbd346" /></button>}
{lastUpdateError && lastUpdateError.date
@@ -114,7 +115,7 @@ const Keyword = (props: KeywordProps) => {
<div
className={`keyword_position absolute bg-[#f8f9ff] w-fit min-w-[50px] h-12 p-2 text-base mt-[-20px] rounded right-5 lg:relative
lg:bg-transparent lg:w-auto lg:h-auto lg:mt-0 lg:p-0 lg:text-sm lg:flex-1 lg:basis-24 lg:grow-0 lg:right-0 text-center font-semibold`}>
<KeywordPosition position={position} />
<KeywordPosition position={position} updating={updating} />
{!updating && positionChange > 0 && <i className=' not-italic ml-1 text-xs text-[#5ed7c3]'> {positionChange}</i>}
{!updating && positionChange < 0 && <i className=' not-italic ml-1 text-xs text-red-300'> {positionChange}</i>}
</div>

View File

@@ -23,7 +23,7 @@ const KeywordFilters = (props: KeywordFilterProps) => {
setDevice,
filterKeywords,
allTags = [],
keywords,
keywords = [],
updateSort,
sortBy,
filterParams,
@@ -35,10 +35,17 @@ const KeywordFilters = (props: KeywordFilterProps) => {
const [filterOptions, showFilterOptions] = useState(false);
const keywordCounts = useMemo(() => {
return keywords.reduce((acc, k) => ({
desktop: k.device === 'desktop' ? acc.desktop + 1 : acc.desktop,
mobile: k.device !== 'desktop' ? acc.mobile + 1 : acc.mobile,
}), { desktop: 0, mobile: 0 });
const counts = { desktop: 0, mobile: 0 };
if (keywords && keywords.length > 0) {
keywords.forEach((k) => {
if (k.device === 'desktop') {
counts.desktop += 1;
} else {
counts.mobile += 1;
}
});
}
return counts;
}, [keywords]);
const filterCountry = (cntrs:string[]) => filterKeywords({ ...filterParams, countries: cntrs });

View File

@@ -1,8 +1,6 @@
import React, { useState, useMemo } from 'react';
import { Toaster } from 'react-hot-toast';
import { CSSTransition } from 'react-transition-group';
import { FixedSizeList as List, ListChildComponentProps } from 'react-window';
import AddKeywords from './AddKeywords';
import { filterKeywords, keywordsByDevice, sortKeywords } from '../../utils/client/sortFilter';
import Icon from '../common/Icon';
import Keyword from './Keyword';
@@ -25,7 +23,7 @@ type KeywordsTableProps = {
}
const KeywordsTable = (props: KeywordsTableProps) => {
const { domain, keywords = [], isLoading = true, showAddModal = false, setShowAddModal, isConsoleIntegrated = false } = props;
const { keywords = [], isLoading = true, isConsoleIntegrated = false } = props;
const showSCData = isConsoleIntegrated;
const [device, setDevice] = useState<string>('desktop');
const [selectedKeywords, setSelectedKeywords] = useState<number[]>([]);
@@ -243,13 +241,6 @@ const KeywordsTable = (props: KeywordsTableProps) => {
</div>
</Modal>
)}
<CSSTransition in={showAddModal} timeout={300} classNames="modal_anim" unmountOnExit mountOnEnter>
<AddKeywords
domain={domain?.domain || ''}
keywords={keywords}
closeModal={() => setShowAddModal(false)}
/>
</CSSTransition>
{showTagManager && (
<KeywordTagManager
allTags={allDomainTags}

View File

@@ -31,7 +31,7 @@ const SCKeywordsTable = ({ domain, keywords = [], isLoading = true, isConsoleInt
const [filterParams, setFilterParams] = useState<KeywordFilters>({ countries: [], tags: [], search: '' });
const [sortBy, setSortBy] = useState<string>('imp_desc');
const [SCListHeight, setSCListHeight] = useState(500);
const { keywordsData } = useFetchKeywords(router);
const { keywordsData } = useFetchKeywords(router, domain?.domain || '');
const addedkeywords: string[] = keywordsData?.keywords?.map((key: KeywordType) => `${key.keyword}:${key.country}:${key.device}`) || [];
const { mutate: addKeywords } = useAddKeywords(() => { if (domain && domain.slug) router.push(`/domain/${domain.slug}`); });
const [isMobile] = useIsMobile();
@@ -209,7 +209,7 @@ const SCKeywordsTable = ({ domain, keywords = [], isLoading = true, isConsoleInt
)}
{!isConsoleIntegrated && (
<p className=' p-9 pt-[10%] text-center text-gray-500'>
Google Search has not been Integrated yet. Please follow <a className='text-indigo-600 underline' href='https://docs.serpbear.com/miscellaneous/integrate-google-search-console' target="_blank" rel='noreferrer'>These Steps</a> to integrate Google Search Data for this Domain.
Google Search Console has not been Integrated yet. Please follow <a className='text-indigo-600 underline' href='https://docs.serpbear.com/miscellaneous/integrate-google-search-console' target="_blank" rel='noreferrer'>These Steps</a> to integrate Google Search Data for this Domain.
</p>
)}
</div>

View File

@@ -1,6 +1,7 @@
import React from 'react';
import SelectField from '../common/SelectField';
import SecretField from '../common/SecretField';
import InputField from '../common/InputField';
type NotificationSettingsProps = {
settings: SettingsType,
@@ -18,8 +19,8 @@ const NotificationSettings = ({ settings, settingsError, updateSettings }:Notifi
<div>
<div className='settings__content styled-scrollbar p-6 text-sm'>
<div className="settings__section__input mb-5">
<label className={labelStyle}>Notification Frequency</label>
<SelectField
label='Notification Frequency'
multiple={false}
selected={[settings.notification_interval]}
options={[
@@ -32,68 +33,61 @@ const NotificationSettings = ({ settings, settingsError, updateSettings }:Notifi
updateField={(updated:string[]) => updated[0] && updateSettings('notification_interval', updated[0])}
rounded='rounded'
maxHeight={48}
minWidth={270}
minWidth={220}
/>
</div>
{settings.notification_interval !== 'never' && (
<>
<div className="settings__section__input mb-5">
<label className={labelStyle}>Notification Emails</label>
<input
className={`w-full p-2 border border-gray-200 rounded mb-3 focus:outline-none focus:border-blue-200
${settingsError?.type === 'no_email' ? ' border-red-400 focus:border-red-400' : ''} `}
type="text"
value={settings?.notification_email}
placeholder={'test@gmail.com'}
onChange={(event) => updateSettings('notification_email', event.target.value)}
<InputField
label='Notification Emails'
hasError={settingsError?.type === 'no_email'}
value={settings?.notification_email}
placeholder={'test@gmail.com, test2@test.com'}
onChange={(value:string) => updateSettings('notification_email', value)}
/>
</div>
<div className="settings__section__input mb-5">
<label className={labelStyle}>SMTP Server</label>
<input
className={`w-full p-2 border border-gray-200 rounded mb-3 focus:outline-none focus:border-blue-200
${settingsError?.type === 'no_smtp_server' ? ' border-red-400 focus:border-red-400' : ''} `}
type="text"
value={settings?.smtp_server || ''}
onChange={(event) => updateSettings('smtp_server', event.target.value)}
<InputField
label='SMTP Server'
hasError={settingsError?.type === 'no_smtp_server'}
value={settings?.smtp_server || ''}
placeholder={'test@gmail.com, test2@test.com'}
onChange={(value:string) => updateSettings('smtp_server', value)}
/>
</div>
<div className="settings__section__input mb-5">
<label className={labelStyle}>SMTP Port</label>
<input
className={`w-full p-2 border border-gray-200 rounded mb-3 focus:outline-none focus:border-blue-200
${settingsError && settingsError.type === 'no_smtp_port' ? ' border-red-400 focus:border-red-400' : ''} `}
type="text"
value={settings?.smtp_port || ''}
onChange={(event) => updateSettings('smtp_port', event.target.value)}
<InputField
label='SMTP Port'
hasError={settingsError?.type === 'no_smtp_port'}
value={settings?.smtp_port || ''}
placeholder={'2234'}
onChange={(value:string) => updateSettings('smtp_port', value)}
/>
</div>
<div className="settings__section__input mb-5">
<label className={labelStyle}>SMTP Username</label>
<input
className={'w-full p-2 border border-gray-200 rounded mb-3 focus:outline-none focus:border-blue-200'}
type="text"
<InputField
label='SMTP Username'
hasError={settingsError?.type === 'no_smtp_port'}
value={settings?.smtp_username || ''}
onChange={(event) => updateSettings('smtp_username', event.target.value)}
onChange={(value:string) => updateSettings('smtp_username', value)}
/>
</div>
<div className="settings__section__input mb-5">
<SecretField
label='SMTP Password'
value={settings?.smtp_password || ''}
onChange={(value:string) => updateSettings('smtp_password', value)}
/>
</div>
<SecretField
label='SMTP Password'
value={settings?.smtp_password || ''}
onChange={(value:string) => updateSettings('smtp_password', value)}
/>
<div className="settings__section__input mb-5">
<label className={labelStyle}>From Email Address</label>
<input
className={`w-full p-2 border border-gray-200 rounded mb-3 focus:outline-none focus:border-blue-200
${settingsError?.type === 'no_smtp_from' ? ' border-red-400 focus:border-red-400' : ''} `}
type="text"
<InputField
label='From Email Address'
hasError={settingsError?.type === 'no_smtp_from'}
value={settings?.notification_email_from || ''}
placeholder="no-reply@mydomain.com"
onChange={(event) => updateSettings('notification_email_from', event.target.value)}
/>
onChange={(value:string) => updateSettings('notification_email_from', value)}
/>
</div>
</>
)}

View File

@@ -3,6 +3,7 @@ import { useClearFailedQueue } from '../../services/settings';
import Icon from '../common/Icon';
import SelectField, { SelectionOption } from '../common/SelectField';
import SecretField from '../common/SecretField';
import ToggleField from '../common/ToggleField';
type ScraperSettingsProps = {
settings: SettingsType,
@@ -44,18 +45,18 @@ const ScraperSettings = ({ settings, settingsError, updateSettings }:ScraperSett
<div className='settings__content styled-scrollbar p-6 text-sm'>
<div className="settings__section__select mb-5">
<label className={labelStyle}>Scraping Method</label>
<SelectField
label='Scraping Method'
options={scraperOptions}
selected={[settings.scraper_type || 'none']}
defaultLabel="Select Scraper"
updateField={(updatedTime:[string]) => updateSettings('scraper_type', updatedTime[0])}
multiple={false}
rounded={'rounded'}
minWidth={270}
minWidth={220}
/>
</div>
{['scrapingant', 'scrapingrobot', 'serply', 'serpapi', 'spaceSerp', 'searchapi'].includes(settings.scraper_type) && (
{settings.scraper_type !== 'none' && settings.scraper_type !== 'proxy' && (
<SecretField
label='Scraper API Key or Token'
placeholder={'API Key/Token'}
@@ -79,8 +80,8 @@ const ScraperSettings = ({ settings, settingsError, updateSettings }:ScraperSett
)}
{settings.scraper_type !== 'none' && (
<div className="settings__section__input mb-5">
<label className={labelStyle}>Scraping Frequency</label>
<SelectField
label='Scraping Frequency'
multiple={false}
selected={[settings?.scrape_interval || 'daily']}
options={scrapingOptions}
@@ -88,14 +89,14 @@ const ScraperSettings = ({ settings, settingsError, updateSettings }:ScraperSett
updateField={(updated:string[]) => updated[0] && updateSettings('scrape_interval', updated[0])}
rounded='rounded'
maxHeight={48}
minWidth={270}
minWidth={220}
/>
<small className=' text-gray-500 pt-2 block'>This option requires Server/Docker Instance Restart to take Effect.</small>
</div>
)}
<div className="settings__section__input mb-5">
<label className={labelStyle}>Delay Between Each keyword Scrape</label>
<SelectField
label='keyword Scrape Delay'
multiple={false}
selected={[settings?.scrape_delay || '0']}
options={delayOptions}
@@ -103,28 +104,16 @@ const ScraperSettings = ({ settings, settingsError, updateSettings }:ScraperSett
updateField={(updated:string[]) => updated[0] && updateSettings('scrape_delay', updated[0])}
rounded='rounded'
maxHeight={48}
minWidth={270}
minWidth={220}
/>
<small className=' text-gray-500 pt-2 block'>This option requires Server/Docker Instance Restart to take Effect.</small>
</div>
<div className="settings__section__input mb-5">
<label className="relative inline-flex items-center cursor-pointer w-full justify-between">
<span className="text-sm font-medium text-gray-900 dark:text-gray-300 w-56">Auto Retry Failed Keyword Scrape</span>
<input
type="checkbox"
value={settings?.scrape_retry ? 'true' : '' }
checked={settings.scrape_retry || false}
className="sr-only peer"
onChange={() => updateSettings('scrape_retry', !settings.scrape_retry)}
/>
<div className="relative rounded-3xl w-9 h-5 bg-gray-200 peer-focus:outline-none peer-focus:ring-4
peer-focus:ring-blue-300 dark:peer-focus:ring-blue-800rounded-full peer dark:bg-gray-700
peer-checked:after:translate-x-full peer-checked:after:border-white after:content-['']
after:absolute after:top-[2px] after:left-[2px] after:bg-white after:border-gray-300
after:border after:rounded-full after:h-4 after:w-4
after:transition-all dark:border-gray-600 peer-checked:bg-blue-600"></div>
</label>
<ToggleField
label='Auto Retry Failed Keyword Scrape'
value={settings?.scrape_retry ? 'true' : '' }
onChange={(val) => updateSettings('scrape_retry', val)}
/>
</div>
{settings?.scrape_retry && (settings.failed_queue?.length || 0) > 0 && (
<div className="settings__section__input mb-5">

View File

@@ -0,0 +1,49 @@
import React from 'react';
import ToggleField from '../common/ToggleField';
import InputField from '../common/InputField';
type SearchConsoleSettingsProps = {
settings: SettingsType,
settingsError: null | {
type: string,
msg: string
},
updateSettings: Function,
}
const SearchConsoleSettings = ({ settings, settingsError, updateSettings }:SearchConsoleSettingsProps) => {
return (
<div>
<div className='settings__content styled-scrollbar p-6 text-sm'>
{/* <div className="settings__section__input mb-5">
<ToggleField
label='Enable Goolge Search Console'
value={settings?.scrape_retry ? 'true' : '' }
onChange={(val) => updateSettings('scrape_retry', val)}
/>
</div> */}
<div className="settings__section__input mb-4 flex justify-between items-center w-full">
<InputField
label='Search Console Client Email'
onChange={(client_email:string) => updateSettings('search_console_client_email', client_email)}
value={settings.search_console_client_email}
placeholder='myapp@appspot.gserviceaccount.com'
/>
</div>
<div className="settings__section__input mb-4 flex flex-col justify-between items-center w-full">
<label className='mb-2 font-semibold block text-sm text-gray-700 capitalize w-full'>Search Console Private Key</label>
<textarea
className={`w-full p-2 border border-gray-200 rounded mb-3 text-xs
focus:outline-none h-[100px] focus:border-blue-200`}
value={settings.search_console_private_key}
placeholder={'-----BEGIN PRIVATE KEY-----/ssssaswdkihad....'}
onChange={(event) => updateSettings('search_console_private_key', event.target.value)}
/>
</div>
</div>
</div>
);
};
export default SearchConsoleSettings;

View File

@@ -5,6 +5,7 @@ import Icon from '../common/Icon';
import NotificationSettings from './NotificationSettings';
import ScraperSettings from './ScraperSettings';
import useOnKey from '../../hooks/useOnKey';
import SearchConsoleSettings from './SearchConsoleSettings';
type SettingsProps = {
closeSettings: Function,
@@ -16,7 +17,7 @@ type SettingsError = {
msg: string
}
const defaultSettings = {
const defaultSettings: SettingsType = {
scraper_type: 'none',
scrape_delay: 'none',
scrape_retry: false,
@@ -27,6 +28,9 @@ const defaultSettings = {
smtp_username: '',
smtp_password: '',
notification_email_from: '',
search_console: true,
search_console_client_email: '',
search_console_private_key: '',
};
const Settings = ({ closeSettings }:SettingsProps) => {
@@ -81,30 +85,38 @@ const Settings = ({ closeSettings }:SettingsProps) => {
}
};
const tabStyle = 'inline-block px-4 py-1 rounded-full mr-3 cursor-pointer text-sm';
const tabStyle = `inline-block px-3 py-2 rounded-md cursor-pointer text-xs lg:text-sm lg:mr-3 lg:px-4 select-none z-10
text-gray-600 border border-b-0 relative top-[1px] rounded-b-none`;
const tabStyleActive = 'bg-white text-blue-600 border-slate-200';
return (
<div className="settings fixed w-full h-screen top-0 left-0 z-50" onClick={closeOnBGClick}>
<div className="absolute w-full max-w-xs bg-white customShadow top-0 right-0 h-screen" data-loading={isLoading} >
<div className="absolute w-full max-w-md bg-white customShadow top-0 right-0 h-screen" data-loading={isLoading} >
{isLoading && <div className='absolute flex content-center items-center h-full'><Icon type="loading" size={24} /></div>}
<div className='settings__header p-6 border-b border-b-slate-200 text-slate-500'>
<div className='settings__header px-5 py-4 text-slate-500'>
<h3 className=' text-black text-lg font-bold'>Settings</h3>
<button
className=' absolute top-2 right-2 p-2 px-3 text-gray-400 hover:text-gray-700 transition-all hover:rotate-90'
className=' absolute top-2 right-2 p-2 px- text-gray-400 hover:text-gray-700 transition-all hover:rotate-90'
onClick={() => closeSettings()}>
<Icon type='close' size={24} />
</button>
</div>
<div className=' px-4 mt-4 '>
<div className='border border-slate-200 px-3 py-4 pb-0 border-l-0 border-r-0 bg-[#f8f9ff]'>
<ul>
<li
className={`${tabStyle} ${currentTab === 'scraper' ? ' bg-blue-50 text-blue-600' : ''}`}
className={`${tabStyle} ${currentTab === 'scraper' ? tabStyleActive : 'border-transparent '}`}
onClick={() => setCurrentTab('scraper')}>
Scraper
<Icon type='scraper' /> Scraper
</li>
<li
className={`${tabStyle} ${currentTab === 'notification' ? ' bg-blue-50 text-blue-600' : ''}`}
className={`${tabStyle} ${currentTab === 'notification' ? tabStyleActive : 'border-transparent'}`}
onClick={() => setCurrentTab('notification')}>
Notification
<Icon type='email' /> Notification
</li>
<li
className={`${tabStyle} ${currentTab === 'searchconsole' ? tabStyleActive : 'border-transparent'}`}
onClick={() => setCurrentTab('searchconsole')}>
<Icon type='google' size={14} /> Search Console
</li>
</ul>
</div>
@@ -115,6 +127,9 @@ const Settings = ({ closeSettings }:SettingsProps) => {
{currentTab === 'notification' && settings && (
<NotificationSettings settings={settings} updateSettings={updateSettings} settingsError={settingsError} />
)}
{currentTab === 'searchconsole' && settings && (
<SearchConsoleSettings settings={settings} updateSettings={updateSettings} settingsError={settingsError} />
)}
<div className=' border-t-[1px] border-gray-200 p-2 px-3'>
<button
onClick={() => performUpdate()}

14
database/config.js Normal file
View File

@@ -0,0 +1,14 @@
module.exports = {
production: {
username: process.env.USER_NAME ? process.env.USER_NAME : process.env.USER,
password: process.env.PASSWORD,
database: 'sequelize',
host: 'database',
port: 3306,
dialect: 'sqlite',
storage: './data/database.sqlite',
dialectOptions: {
bigNumberStrings: true,
},
},
};

View File

@@ -0,0 +1,19 @@
// Migration: Adds city, latlong and settings keyword to keyword table.
// CLI Migration
module.exports = {
up: (queryInterface, Sequelize) => {
return queryInterface.sequelize.transaction(async (t) => {
await queryInterface.addColumn('keyword', 'city', { type: Sequelize.DataTypes.STRING }, { transaction: t });
await queryInterface.addColumn('keyword', 'latlong', { type: Sequelize.DataTypes.STRING }, { transaction: t });
await queryInterface.addColumn('keyword', 'settings', { type: Sequelize.DataTypes.STRING }, { transaction: t });
});
},
down: (queryInterface) => {
return queryInterface.sequelize.transaction(async (t) => {
await queryInterface.removeColumn('keyword', 'city', { transaction: t });
await queryInterface.removeColumn('keyword', 'latlong', { transaction: t });
await queryInterface.removeColumn('keyword', 'settings', { transaction: t });
});
},
};

View File

@@ -0,0 +1,15 @@
// Migration: Adds search_console field to domain table to assign search console property type, url and api.
// CLI Migration
module.exports = {
up: (queryInterface, Sequelize) => {
return queryInterface.sequelize.transaction(async (t) => {
await queryInterface.addColumn('domain', 'search_console', { type: Sequelize.DataTypes.STRING }, { transaction: t });
});
},
down: (queryInterface) => {
return queryInterface.sequelize.transaction(async (t) => {
await queryInterface.removeColumn('domain', 'search_console', { transaction: t });
});
},
};

View File

@@ -38,6 +38,9 @@ class Domain extends Model {
@Column({ type: DataType.STRING, allowNull: true, defaultValue: '' })
notification_emails!: string;
@Column({ type: DataType.STRING, allowNull: true })
search_console!: string;
}
export default Domain;

View File

@@ -19,7 +19,13 @@ class Keyword extends Model {
@Column({ type: DataType.STRING, allowNull: true, defaultValue: 'US' })
country!: string;
@Column({ type: DataType.STRING, allowNull: false })
@Column({ type: DataType.STRING, allowNull: true, defaultValue: '' })
city!: string;
@Column({ type: DataType.STRING, allowNull: true, defaultValue: '' })
latlong!: string;
@Column({ type: DataType.STRING, allowNull: false, defaultValue: '{}' })
domain!: string;
// @ForeignKey(() => Domain)
@@ -58,6 +64,9 @@ class Keyword extends Model {
@Column({ type: DataType.STRING, allowNull: true, defaultValue: 'false' })
lastUpdateError!: string;
@Column({ type: DataType.STRING, allowNull: true })
settings!: string;
}
export default Keyword;

3
entrypoint.sh Normal file
View File

@@ -0,0 +1,3 @@
#!/bin/sh
npx sequelize-cli db:migrate --env production
exec "$@"

756
package-lock.json generated
View File

@@ -1,12 +1,12 @@
{
"name": "serpbear",
"version": "0.3.4",
"version": "1.0.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "serpbear",
"version": "0.3.4",
"version": "1.0.0",
"dependencies": {
"@googleapis/searchconsole": "^1.0.0",
"@types/react-transition-group": "^4.4.5",
@@ -25,7 +25,7 @@
"jsonwebtoken": "^9.0.2",
"msw": "^0.49.0",
"next": "^12.3.4",
"nodemailer": "^6.8.0",
"nodemailer": "^6.9.9",
"react": "18.2.0",
"react-chartjs-2": "^4.3.1",
"react-dom": "18.2.0",
@@ -36,7 +36,7 @@
"react-window": "^1.8.8",
"reflect-metadata": "^0.1.13",
"sequelize": "^6.34.0",
"sequelize-typescript": "^2.1.5",
"sequelize-typescript": "^2.1.6",
"sqlite3": "^5.1.6"
},
"devDependencies": {
@@ -65,6 +65,7 @@
"prettier": "^2.7.1",
"resize-observer-polyfill": "^1.5.1",
"sass": "^1.55.0",
"sequelize-cli": "^6.6.2",
"standard-version": "^9.5.0",
"stylelint-config-standard": "^29.0.0",
"tailwindcss": "^3.1.8",
@@ -852,6 +853,96 @@
"node": ">=6.9.0"
}
},
"node_modules/@isaacs/cliui": {
"version": "8.0.2",
"resolved": "https://registry.npmjs.org/@isaacs/cliui/-/cliui-8.0.2.tgz",
"integrity": "sha512-O8jcjabXaleOG9DQ0+ARXWZBTfnP4WNAqzuiJK7ll44AmxGKv/J2M4TPjxjY3znBCfvBXFzucm1twdyFybFqEA==",
"dev": true,
"dependencies": {
"string-width": "^5.1.2",
"string-width-cjs": "npm:string-width@^4.2.0",
"strip-ansi": "^7.0.1",
"strip-ansi-cjs": "npm:strip-ansi@^6.0.1",
"wrap-ansi": "^8.1.0",
"wrap-ansi-cjs": "npm:wrap-ansi@^7.0.0"
},
"engines": {
"node": ">=12"
}
},
"node_modules/@isaacs/cliui/node_modules/ansi-regex": {
"version": "6.0.1",
"resolved": "https://registry.npmjs.org/ansi-regex/-/ansi-regex-6.0.1.tgz",
"integrity": "sha512-n5M855fKb2SsfMIiFFoVrABHJC8QtHwVx+mHWP3QcEqBHYienj5dHSgjbxtC0WEZXYt4wcD6zrQElDPhFuZgfA==",
"dev": true,
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://github.com/chalk/ansi-regex?sponsor=1"
}
},
"node_modules/@isaacs/cliui/node_modules/ansi-styles": {
"version": "6.2.1",
"resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-6.2.1.tgz",
"integrity": "sha512-bN798gFfQX+viw3R7yrGWRqnrN2oRkEkUjjl4JNn4E8GxxbjtG3FbrEIIY3l8/hrwUwIeCZvi4QuOTP4MErVug==",
"dev": true,
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://github.com/chalk/ansi-styles?sponsor=1"
}
},
"node_modules/@isaacs/cliui/node_modules/string-width": {
"version": "5.1.2",
"resolved": "https://registry.npmjs.org/string-width/-/string-width-5.1.2.tgz",
"integrity": "sha512-HnLOCR3vjcY8beoNLtcjZ5/nxn2afmME6lhrDrebokqMap+XbeW8n9TXpPDOqdGK5qcI3oT0GKTW6wC7EMiVqA==",
"dev": true,
"dependencies": {
"eastasianwidth": "^0.2.0",
"emoji-regex": "^9.2.2",
"strip-ansi": "^7.0.1"
},
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/@isaacs/cliui/node_modules/strip-ansi": {
"version": "7.1.0",
"resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-7.1.0.tgz",
"integrity": "sha512-iq6eVVI64nQQTRYq2KtEg2d2uU7LElhTJwsH4YzIHZshxlgZms/wIc4VoDQTlG/IvVIrBKG06CrZnp0qv7hkcQ==",
"dev": true,
"dependencies": {
"ansi-regex": "^6.0.1"
},
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://github.com/chalk/strip-ansi?sponsor=1"
}
},
"node_modules/@isaacs/cliui/node_modules/wrap-ansi": {
"version": "8.1.0",
"resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-8.1.0.tgz",
"integrity": "sha512-si7QWI6zUMq56bESFvagtmzMdGOtoxfR+Sez11Mobfc7tm+VkUckk9bW2UeffTGVUbOksxmSw0AA2gs8g71NCQ==",
"dev": true,
"dependencies": {
"ansi-styles": "^6.1.0",
"string-width": "^5.0.1",
"strip-ansi": "^7.0.1"
},
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://github.com/chalk/wrap-ansi?sponsor=1"
}
},
"node_modules/@istanbuljs/load-nyc-config": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/@istanbuljs/load-nyc-config/-/load-nyc-config-1.1.0.tgz",
@@ -1813,11 +1904,27 @@
"node": ">=10"
}
},
"node_modules/@one-ini/wasm": {
"version": "0.1.1",
"resolved": "https://registry.npmjs.org/@one-ini/wasm/-/wasm-0.1.1.tgz",
"integrity": "sha512-XuySG1E38YScSJoMlqovLru4KTUNSjgVTIjyh7qMX6aNN5HY5Ct5LhRJdxO79JtTzKfzV/bnWpz+zquYrISsvw==",
"dev": true
},
"node_modules/@open-draft/until": {
"version": "1.0.3",
"resolved": "https://registry.npmjs.org/@open-draft/until/-/until-1.0.3.tgz",
"integrity": "sha512-Aq58f5HiWdyDlFffbbSjAlv596h/cOnt2DO1w3DOC7OJ5EHs0hd/nycJfiu9RJbT6Yk6F1knnRRXNSpxoIVZ9Q=="
},
"node_modules/@pkgjs/parseargs": {
"version": "0.11.0",
"resolved": "https://registry.npmjs.org/@pkgjs/parseargs/-/parseargs-0.11.0.tgz",
"integrity": "sha512-+1VkjdD0QBLPodGrJUeqarH8VAIvQODIbwh9XpP5Syisf7YoQgsJKPNFoqqLQlu+VQ/tVSshMR6loPMn8U+dPg==",
"dev": true,
"optional": true,
"engines": {
"node": ">=14"
}
},
"node_modules/@rushstack/eslint-patch": {
"version": "1.5.1",
"resolved": "https://registry.npmjs.org/@rushstack/eslint-patch/-/eslint-patch-1.5.1.tgz",
@@ -2920,6 +3027,15 @@
"resolved": "https://registry.npmjs.org/asynckit/-/asynckit-0.4.0.tgz",
"integrity": "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q=="
},
"node_modules/at-least-node": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/at-least-node/-/at-least-node-1.0.0.tgz",
"integrity": "sha512-+q/t7Ekv1EDY2l6Gda6LLiX14rU9TV20Wa3ofeQmwPFZbOMo9DXrLbOjFaaclkXKWidIaopwAObQDqwWtGUjqg==",
"dev": true,
"engines": {
"node": ">= 4.0.0"
}
},
"node_modules/autoprefixer": {
"version": "10.4.16",
"resolved": "https://registry.npmjs.org/autoprefixer/-/autoprefixer-10.4.16.tgz",
@@ -3186,6 +3302,12 @@
"readable-stream": "^3.4.0"
}
},
"node_modules/bluebird": {
"version": "3.7.2",
"resolved": "https://registry.npmjs.org/bluebird/-/bluebird-3.7.2.tgz",
"integrity": "sha512-XpNj6GDQzdfW+r2Wnn7xiSAd7TM3jzkxGXBGTtWKuSXv1xUV+azxAm8jdWZN06QTQk+2N2XB9jRDkvbmQmcRtg==",
"dev": true
},
"node_modules/boolbase": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/boolbase/-/boolbase-1.0.0.tgz",
@@ -3567,6 +3689,22 @@
"node": ">=6"
}
},
"node_modules/cli-color": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/cli-color/-/cli-color-2.0.3.tgz",
"integrity": "sha512-OkoZnxyC4ERN3zLzZaY9Emb7f/MhBOIpePv0Ycok0fJYT+Ouo00UBEIwsVsr0yoow++n5YWlSUgST9GKhNHiRQ==",
"dev": true,
"dependencies": {
"d": "^1.0.1",
"es5-ext": "^0.10.61",
"es6-iterator": "^2.0.3",
"memoizee": "^0.4.15",
"timers-ext": "^0.1.7"
},
"engines": {
"node": ">=0.10"
}
},
"node_modules/cli-cursor": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/cli-cursor/-/cli-cursor-3.1.0.tgz",
@@ -3797,6 +3935,16 @@
"url": "https://github.com/chalk/supports-color?sponsor=1"
}
},
"node_modules/config-chain": {
"version": "1.1.13",
"resolved": "https://registry.npmjs.org/config-chain/-/config-chain-1.1.13.tgz",
"integrity": "sha512-qj+f8APARXHrM0hraqXYb2/bOVSV4PvJQlNZ/DVj0QrmNM2q2euizkeuVckQ57J+W0mRH6Hvi+k50M4Jul2VRQ==",
"dev": true,
"dependencies": {
"ini": "^1.3.4",
"proto-list": "~1.2.1"
}
},
"node_modules/confusing-browser-globals": {
"version": "1.0.11",
"resolved": "https://registry.npmjs.org/confusing-browser-globals/-/confusing-browser-globals-1.0.11.tgz",
@@ -4265,6 +4413,16 @@
"resolved": "https://registry.npmjs.org/csstype/-/csstype-3.1.2.tgz",
"integrity": "sha512-I7K1Uu0MBPzaFKg4nI5Q7Vs2t+3gWWW648spaF+Rg7pI9ds18Ugn+lvg4SHczUdKlHI5LWBXyqfS8+DufyBsgQ=="
},
"node_modules/d": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/d/-/d-1.0.1.tgz",
"integrity": "sha512-m62ShEObQ39CfralilEQRjH6oAMtNCV1xJyEx5LpRYUVN+EviphDgUc/F3hnYbADmkiNs67Y+3ylmlG7Lnu+FA==",
"dev": true,
"dependencies": {
"es5-ext": "^0.10.50",
"type": "^1.0.1"
}
},
"node_modules/damerau-levenshtein": {
"version": "1.0.8",
"resolved": "https://registry.npmjs.org/damerau-levenshtein/-/damerau-levenshtein-1.0.8.tgz",
@@ -4764,6 +4922,12 @@
"resolved": "https://registry.npmjs.org/dottie/-/dottie-2.0.6.tgz",
"integrity": "sha512-iGCHkfUc5kFekGiqhe8B/mdaurD+lakO9txNnTvKtA6PISrw86LgqHvRzWYPyoE2Ph5aMIrCw9/uko6XHTKCwA=="
},
"node_modules/eastasianwidth": {
"version": "0.2.0",
"resolved": "https://registry.npmjs.org/eastasianwidth/-/eastasianwidth-0.2.0.tgz",
"integrity": "sha512-I88TYZWc9XiYHRQ4/3c5rjjfgkjhLyW2luGIheGERbNQ6OY7yTybanSpDXZa8y7VUP9YmDcYa+eyq4ca7iLqWA==",
"dev": true
},
"node_modules/ecdsa-sig-formatter": {
"version": "1.0.11",
"resolved": "https://registry.npmjs.org/ecdsa-sig-formatter/-/ecdsa-sig-formatter-1.0.11.tgz",
@@ -4772,6 +4936,90 @@
"safe-buffer": "^5.0.1"
}
},
"node_modules/editorconfig": {
"version": "1.0.4",
"resolved": "https://registry.npmjs.org/editorconfig/-/editorconfig-1.0.4.tgz",
"integrity": "sha512-L9Qe08KWTlqYMVvMcTIvMAdl1cDUubzRNYL+WfA4bLDMHe4nemKkpmYzkznE1FwLKu0EEmy6obgQKzMJrg4x9Q==",
"dev": true,
"dependencies": {
"@one-ini/wasm": "0.1.1",
"commander": "^10.0.0",
"minimatch": "9.0.1",
"semver": "^7.5.3"
},
"bin": {
"editorconfig": "bin/editorconfig"
},
"engines": {
"node": ">=14"
}
},
"node_modules/editorconfig/node_modules/brace-expansion": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.0.1.tgz",
"integrity": "sha512-XnAIvQ8eM+kC6aULx6wuQiwVsnzsi9d3WxzV3FpWTGA19F621kwdbsAcFKXgKUHZWsy+mY6iL1sHTxWEFCytDA==",
"dev": true,
"dependencies": {
"balanced-match": "^1.0.0"
}
},
"node_modules/editorconfig/node_modules/commander": {
"version": "10.0.1",
"resolved": "https://registry.npmjs.org/commander/-/commander-10.0.1.tgz",
"integrity": "sha512-y4Mg2tXshplEbSGzx7amzPwKKOCGuoSRP/CjEdwwk0FOGlUbq6lKuoyDZTNZkmxHdJtp54hdfY/JUrdL7Xfdug==",
"dev": true,
"engines": {
"node": ">=14"
}
},
"node_modules/editorconfig/node_modules/lru-cache": {
"version": "6.0.0",
"resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-6.0.0.tgz",
"integrity": "sha512-Jo6dJ04CmSjuznwJSS3pUeWmd/H0ffTlkXXgwZi+eq1UCmqQwCh+eLsYOYCwY991i2Fah4h1BEMCx4qThGbsiA==",
"dev": true,
"dependencies": {
"yallist": "^4.0.0"
},
"engines": {
"node": ">=10"
}
},
"node_modules/editorconfig/node_modules/minimatch": {
"version": "9.0.1",
"resolved": "https://registry.npmjs.org/minimatch/-/minimatch-9.0.1.tgz",
"integrity": "sha512-0jWhJpD/MdhPXwPuiRkCbfYfSKp2qnn2eOc279qI7f+osl/l+prKSrvhg157zSYvx/1nmgn2NqdT6k2Z7zSH9w==",
"dev": true,
"dependencies": {
"brace-expansion": "^2.0.1"
},
"engines": {
"node": ">=16 || 14 >=14.17"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/editorconfig/node_modules/semver": {
"version": "7.5.4",
"resolved": "https://registry.npmjs.org/semver/-/semver-7.5.4.tgz",
"integrity": "sha512-1bCSESV6Pv+i21Hvpxp3Dx+pSD8lIPt8uVjRrxAUt/nbswYc+tK6Y2btiULjd4+fnq15PX+nqQDC7Oft7WkwcA==",
"dev": true,
"dependencies": {
"lru-cache": "^6.0.0"
},
"bin": {
"semver": "bin/semver.js"
},
"engines": {
"node": ">=10"
}
},
"node_modules/editorconfig/node_modules/yallist": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/yallist/-/yallist-4.0.0.tgz",
"integrity": "sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==",
"dev": true
},
"node_modules/electron-to-chromium": {
"version": "1.4.576",
"resolved": "https://registry.npmjs.org/electron-to-chromium/-/electron-to-chromium-1.4.576.tgz",
@@ -4987,6 +5235,54 @@
"url": "https://github.com/sponsors/ljharb"
}
},
"node_modules/es5-ext": {
"version": "0.10.62",
"resolved": "https://registry.npmjs.org/es5-ext/-/es5-ext-0.10.62.tgz",
"integrity": "sha512-BHLqn0klhEpnOKSrzn/Xsz2UIW8j+cGmo9JLzr8BiUapV8hPL9+FliFqjwr9ngW7jWdnxv6eO+/LqyhJVqgrjA==",
"dev": true,
"hasInstallScript": true,
"dependencies": {
"es6-iterator": "^2.0.3",
"es6-symbol": "^3.1.3",
"next-tick": "^1.1.0"
},
"engines": {
"node": ">=0.10"
}
},
"node_modules/es6-iterator": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/es6-iterator/-/es6-iterator-2.0.3.tgz",
"integrity": "sha512-zw4SRzoUkd+cl+ZoE15A9o1oQd920Bb0iOJMQkQhl3jNc03YqVjAhG7scf9C5KWRU/R13Orf588uCC6525o02g==",
"dev": true,
"dependencies": {
"d": "1",
"es5-ext": "^0.10.35",
"es6-symbol": "^3.1.1"
}
},
"node_modules/es6-symbol": {
"version": "3.1.3",
"resolved": "https://registry.npmjs.org/es6-symbol/-/es6-symbol-3.1.3.tgz",
"integrity": "sha512-NJ6Yn3FuDinBaBRWl/q5X/s4koRHBrgKAu+yGI6JCBeiu3qrcbJhwT2GeR/EXVfylRk8dpQVJoLEFhK+Mu31NA==",
"dev": true,
"dependencies": {
"d": "^1.0.1",
"ext": "^1.1.2"
}
},
"node_modules/es6-weak-map": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/es6-weak-map/-/es6-weak-map-2.0.3.tgz",
"integrity": "sha512-p5um32HOTO1kP+w7PRnB+5lQ43Z6muuMuIMffvDN8ZB4GcnjLBV6zGStpbASIMk4DCAvEaamhe2zhyCb/QXXsA==",
"dev": true,
"dependencies": {
"d": "1",
"es5-ext": "^0.10.46",
"es6-iterator": "^2.0.3",
"es6-symbol": "^3.1.1"
}
},
"node_modules/escalade": {
"version": "3.1.1",
"resolved": "https://registry.npmjs.org/escalade/-/escalade-3.1.1.tgz",
@@ -5510,6 +5806,16 @@
"node": ">=0.10.0"
}
},
"node_modules/event-emitter": {
"version": "0.3.5",
"resolved": "https://registry.npmjs.org/event-emitter/-/event-emitter-0.3.5.tgz",
"integrity": "sha512-D9rRn9y7kLPnJ+hMq7S/nhvoKwwvVJahBi2BPmx3bvbsEdK3W9ii8cBSGjP+72/LnM4n6fo3+dkCX5FeTQruXA==",
"dev": true,
"dependencies": {
"d": "1",
"es5-ext": "~0.10.14"
}
},
"node_modules/events": {
"version": "3.3.0",
"resolved": "https://registry.npmjs.org/events/-/events-3.3.0.tgz",
@@ -5566,6 +5872,21 @@
"node": "^14.15.0 || ^16.10.0 || >=18.0.0"
}
},
"node_modules/ext": {
"version": "1.7.0",
"resolved": "https://registry.npmjs.org/ext/-/ext-1.7.0.tgz",
"integrity": "sha512-6hxeJYaL110a9b5TEJSj0gojyHQAmA2ch5Os+ySCiA1QGdS697XWY1pzsrSjqA9LDEEgdB/KypIlR59RcLuHYw==",
"dev": true,
"dependencies": {
"type": "^2.7.2"
}
},
"node_modules/ext/node_modules/type": {
"version": "2.7.2",
"resolved": "https://registry.npmjs.org/type/-/type-2.7.2.tgz",
"integrity": "sha512-dzlvlNlt6AXU7EBSfpAscydQ7gXB+pPGsPnfJnZpiNJBDj7IaJzQlBZYGdEi4R9HmPdBv2XmWJ6YUtoTa7lmCw==",
"dev": true
},
"node_modules/extend": {
"version": "3.0.2",
"resolved": "https://registry.npmjs.org/extend/-/extend-3.0.2.tgz",
@@ -5766,6 +6087,34 @@
"is-callable": "^1.1.3"
}
},
"node_modules/foreground-child": {
"version": "3.1.1",
"resolved": "https://registry.npmjs.org/foreground-child/-/foreground-child-3.1.1.tgz",
"integrity": "sha512-TMKDUnIte6bfb5nWv7V/caI169OHgvwjb7V4WkeUvbQQdjr5rWKqHFiKWb/fcOwB+CzBT+qbWjvj+DVwRskpIg==",
"dev": true,
"dependencies": {
"cross-spawn": "^7.0.0",
"signal-exit": "^4.0.1"
},
"engines": {
"node": ">=14"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/foreground-child/node_modules/signal-exit": {
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/signal-exit/-/signal-exit-4.1.0.tgz",
"integrity": "sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw==",
"dev": true,
"engines": {
"node": ">=14"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/form-data": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.0.tgz",
@@ -5792,6 +6141,30 @@
"url": "https://github.com/sponsors/rawify"
}
},
"node_modules/fs-extra": {
"version": "9.1.0",
"resolved": "https://registry.npmjs.org/fs-extra/-/fs-extra-9.1.0.tgz",
"integrity": "sha512-hcg3ZmepS30/7BSFqRvoo3DOMQu7IjqxO5nCDt+zM9XWjb33Wg7ziNT+Qvqbuc3+gWpzO02JubVyk2G4Zvo1OQ==",
"dev": true,
"dependencies": {
"at-least-node": "^1.0.0",
"graceful-fs": "^4.2.0",
"jsonfile": "^6.0.1",
"universalify": "^2.0.0"
},
"engines": {
"node": ">=10"
}
},
"node_modules/fs-extra/node_modules/universalify": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/universalify/-/universalify-2.0.1.tgz",
"integrity": "sha512-gptHNQghINnc/vTGIk0SOFGFNXw7JVrlRUtConJRlvaw6DuX0wO5Jeko9sWrMBhh+PsYAZ7oXAiOnf/UKogyiw==",
"dev": true,
"engines": {
"node": ">= 10.0.0"
}
},
"node_modules/fs-minipass": {
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/fs-minipass/-/fs-minipass-2.1.0.tgz",
@@ -7103,6 +7476,12 @@
"integrity": "sha512-bCYeRA2rVibKZd+s2625gGnGF/t7DSqDs4dP7CrLA1m7jKWz6pps0LpYLJN8Q64HtmPKJ1hrN3nzPNKFEKOUiQ==",
"dev": true
},
"node_modules/is-promise": {
"version": "2.2.2",
"resolved": "https://registry.npmjs.org/is-promise/-/is-promise-2.2.2.tgz",
"integrity": "sha512-+lP4/6lKUBfQjZ2pdxThZvLUAafmZb8OAxFb8XXtiQmS35INgr85hdOGoEs124ez1FCnZJt6jau/T+alh58QFQ==",
"dev": true
},
"node_modules/is-regex": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/is-regex/-/is-regex-1.1.4.tgz",
@@ -7396,6 +7775,24 @@
"set-function-name": "^2.0.1"
}
},
"node_modules/jackspeak": {
"version": "2.3.6",
"resolved": "https://registry.npmjs.org/jackspeak/-/jackspeak-2.3.6.tgz",
"integrity": "sha512-N3yCS/NegsOBokc8GAdM8UcmfsKiSS8cipheD/nivzr700H+nsMOxJjQnvwOcRYVuFkdH0wGUvW2WbXGmrZGbQ==",
"dev": true,
"dependencies": {
"@isaacs/cliui": "^8.0.2"
},
"engines": {
"node": ">=14"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
},
"optionalDependencies": {
"@pkgjs/parseargs": "^0.11.0"
}
},
"node_modules/jest": {
"version": "29.7.0",
"resolved": "https://registry.npmjs.org/jest/-/jest-29.7.0.tgz",
@@ -8542,6 +8939,105 @@
"jiti": "bin/jiti.js"
}
},
"node_modules/js-beautify": {
"version": "1.14.11",
"resolved": "https://registry.npmjs.org/js-beautify/-/js-beautify-1.14.11.tgz",
"integrity": "sha512-rPogWqAfoYh1Ryqqh2agUpVfbxAhbjuN1SmU86dskQUKouRiggUTCO4+2ym9UPXllc2WAp0J+T5qxn7Um3lCdw==",
"dev": true,
"dependencies": {
"config-chain": "^1.1.13",
"editorconfig": "^1.0.3",
"glob": "^10.3.3",
"nopt": "^7.2.0"
},
"bin": {
"css-beautify": "js/bin/css-beautify.js",
"html-beautify": "js/bin/html-beautify.js",
"js-beautify": "js/bin/js-beautify.js"
},
"engines": {
"node": ">=14"
}
},
"node_modules/js-beautify/node_modules/abbrev": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/abbrev/-/abbrev-2.0.0.tgz",
"integrity": "sha512-6/mh1E2u2YgEsCHdY0Yx5oW+61gZU+1vXaoiHHrpKeuRNNgFvS+/jrwHiQhB5apAf5oB7UB7E19ol2R2LKH8hQ==",
"dev": true,
"engines": {
"node": "^14.17.0 || ^16.13.0 || >=18.0.0"
}
},
"node_modules/js-beautify/node_modules/brace-expansion": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/brace-expansion/-/brace-expansion-2.0.1.tgz",
"integrity": "sha512-XnAIvQ8eM+kC6aULx6wuQiwVsnzsi9d3WxzV3FpWTGA19F621kwdbsAcFKXgKUHZWsy+mY6iL1sHTxWEFCytDA==",
"dev": true,
"dependencies": {
"balanced-match": "^1.0.0"
}
},
"node_modules/js-beautify/node_modules/glob": {
"version": "10.3.10",
"resolved": "https://registry.npmjs.org/glob/-/glob-10.3.10.tgz",
"integrity": "sha512-fa46+tv1Ak0UPK1TOy/pZrIybNNt4HCv7SDzwyfiOZkvZLEbjsZkJBPtDHVshZjbecAoAGSC20MjLDG/qr679g==",
"dev": true,
"dependencies": {
"foreground-child": "^3.1.0",
"jackspeak": "^2.3.5",
"minimatch": "^9.0.1",
"minipass": "^5.0.0 || ^6.0.2 || ^7.0.0",
"path-scurry": "^1.10.1"
},
"bin": {
"glob": "dist/esm/bin.mjs"
},
"engines": {
"node": ">=16 || 14 >=14.17"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/js-beautify/node_modules/minimatch": {
"version": "9.0.3",
"resolved": "https://registry.npmjs.org/minimatch/-/minimatch-9.0.3.tgz",
"integrity": "sha512-RHiac9mvaRw0x3AYRgDC1CxAP7HTcNrrECeA8YYJeWnpo+2Q5CegtZjaotWTWxDG3UeGA1coE05iH1mPjT/2mg==",
"dev": true,
"dependencies": {
"brace-expansion": "^2.0.1"
},
"engines": {
"node": ">=16 || 14 >=14.17"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/js-beautify/node_modules/minipass": {
"version": "7.0.4",
"resolved": "https://registry.npmjs.org/minipass/-/minipass-7.0.4.tgz",
"integrity": "sha512-jYofLM5Dam9279rdkWzqHozUo4ybjdZmCsDHePy5V/PbBcVMiSZR97gmAy45aqi8CK1lG2ECd356FU86avfwUQ==",
"dev": true,
"engines": {
"node": ">=16 || 14 >=14.17"
}
},
"node_modules/js-beautify/node_modules/nopt": {
"version": "7.2.0",
"resolved": "https://registry.npmjs.org/nopt/-/nopt-7.2.0.tgz",
"integrity": "sha512-CVDtwCdhYIvnAzFoJ6NJ6dX3oga9/HyciQDnG1vQDjSLMeKLJ4A93ZqYKDrgYSr1FBY5/hMYC+2VCi24pgpkGA==",
"dev": true,
"dependencies": {
"abbrev": "^2.0.0"
},
"bin": {
"nopt": "bin/nopt.js"
},
"engines": {
"node": "^14.17.0 || ^16.13.0 || >=18.0.0"
}
},
"node_modules/js-levenshtein": {
"version": "1.1.6",
"resolved": "https://registry.npmjs.org/js-levenshtein/-/js-levenshtein-1.1.6.tgz",
@@ -8695,6 +9191,27 @@
"node": ">=6"
}
},
"node_modules/jsonfile": {
"version": "6.1.0",
"resolved": "https://registry.npmjs.org/jsonfile/-/jsonfile-6.1.0.tgz",
"integrity": "sha512-5dgndWOriYSm5cnYaJNhalLNDKOqFwyDB/rr1E9ZsGciGvKPs8R2xYGCacuf3z6K1YKDz182fd+fY3cn3pMqXQ==",
"dev": true,
"dependencies": {
"universalify": "^2.0.0"
},
"optionalDependencies": {
"graceful-fs": "^4.1.6"
}
},
"node_modules/jsonfile/node_modules/universalify": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/universalify/-/universalify-2.0.1.tgz",
"integrity": "sha512-gptHNQghINnc/vTGIk0SOFGFNXw7JVrlRUtConJRlvaw6DuX0wO5Jeko9sWrMBhh+PsYAZ7oXAiOnf/UKogyiw==",
"dev": true,
"engines": {
"node": ">= 10.0.0"
}
},
"node_modules/jsonparse": {
"version": "1.3.1",
"resolved": "https://registry.npmjs.org/jsonparse/-/jsonparse-1.3.1.tgz",
@@ -9094,6 +9611,15 @@
"yallist": "^3.0.2"
}
},
"node_modules/lru-queue": {
"version": "0.1.0",
"resolved": "https://registry.npmjs.org/lru-queue/-/lru-queue-0.1.0.tgz",
"integrity": "sha512-BpdYkt9EvGl8OfWHDQPISVpcl5xZthb+XPsbELj5AQXxIC8IriDZIQYjBJPEm5rS420sjZ0TLEzRcq5KdBhYrQ==",
"dev": true,
"dependencies": {
"es5-ext": "~0.10.2"
}
},
"node_modules/lz-string": {
"version": "1.5.0",
"resolved": "https://registry.npmjs.org/lz-string/-/lz-string-1.5.0.tgz",
@@ -9265,6 +9791,22 @@
"resolved": "https://registry.npmjs.org/memoize-one/-/memoize-one-5.2.1.tgz",
"integrity": "sha512-zYiwtZUcYyXKo/np96AGZAckk+FWWsUdJ3cHGGmld7+AhvcWmQyGCYUh1hc4Q/pkOhb65dQR/pqCyK0cOaHz4Q=="
},
"node_modules/memoizee": {
"version": "0.4.15",
"resolved": "https://registry.npmjs.org/memoizee/-/memoizee-0.4.15.tgz",
"integrity": "sha512-UBWmJpLZd5STPm7PMUlOw/TSy972M+z8gcyQ5veOnSDRREz/0bmpyTfKt3/51DhEBqCZQn1udM/5flcSPYhkdQ==",
"dev": true,
"dependencies": {
"d": "^1.0.1",
"es5-ext": "^0.10.53",
"es6-weak-map": "^2.0.3",
"event-emitter": "^0.3.5",
"is-promise": "^2.2.2",
"lru-queue": "^0.1.0",
"next-tick": "^1.1.0",
"timers-ext": "^0.1.7"
}
},
"node_modules/meow": {
"version": "8.1.2",
"resolved": "https://registry.npmjs.org/meow/-/meow-8.1.2.tgz",
@@ -9871,6 +10413,12 @@
"react": ">=17.0.0"
}
},
"node_modules/next-tick": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/next-tick/-/next-tick-1.1.0.tgz",
"integrity": "sha512-CXdUiJembsNjuToQvxayPZF9Vqht7hewsvy2sOWafLvi2awflj9mOC6bHIg50orX8IJvWKY9wYQ/zB2kogPslQ==",
"dev": true
},
"node_modules/next/node_modules/postcss": {
"version": "8.4.14",
"resolved": "https://registry.npmjs.org/postcss/-/postcss-8.4.14.tgz",
@@ -10054,9 +10602,9 @@
"dev": true
},
"node_modules/nodemailer": {
"version": "6.9.7",
"resolved": "https://registry.npmjs.org/nodemailer/-/nodemailer-6.9.7.tgz",
"integrity": "sha512-rUtR77ksqex/eZRLmQ21LKVH5nAAsVicAtAYudK7JgwenEDZ0UIQ1adUGqErz7sMkWYxWTTU1aeP2Jga6WQyJw==",
"version": "6.9.9",
"resolved": "https://registry.npmjs.org/nodemailer/-/nodemailer-6.9.9.tgz",
"integrity": "sha512-dexTll8zqQoVJEZPwQAKzxxtFn0qTnjdQTchoU6Re9BUUGBJiOy3YMn/0ShTW6J5M0dfQ1NeDeRTTl4oIWgQMA==",
"engines": {
"node": ">=6.0.0"
}
@@ -10554,6 +11102,40 @@
"integrity": "sha512-LDJzPVEEEPR+y48z93A0Ed0yXb8pAByGWo/k5YYdYgpY2/2EsOsksJrq7lOHxryrVOn1ejG6oAp8ahvOIQD8sw==",
"dev": true
},
"node_modules/path-scurry": {
"version": "1.10.1",
"resolved": "https://registry.npmjs.org/path-scurry/-/path-scurry-1.10.1.tgz",
"integrity": "sha512-MkhCqzzBEpPvxxQ71Md0b1Kk51W01lrYvlMzSUaIzNsODdd7mqhiimSZlr+VegAz5Z6Vzt9Xg2ttE//XBhH3EQ==",
"dev": true,
"dependencies": {
"lru-cache": "^9.1.1 || ^10.0.0",
"minipass": "^5.0.0 || ^6.0.2 || ^7.0.0"
},
"engines": {
"node": ">=16 || 14 >=14.17"
},
"funding": {
"url": "https://github.com/sponsors/isaacs"
}
},
"node_modules/path-scurry/node_modules/lru-cache": {
"version": "10.2.0",
"resolved": "https://registry.npmjs.org/lru-cache/-/lru-cache-10.2.0.tgz",
"integrity": "sha512-2bIM8x+VAf6JT4bKAljS1qUWgMsqZRPGJS6FSahIMPVvctcNhyVp7AJu7quxOW9jwkryBReKZY5tY5JYv2n/7Q==",
"dev": true,
"engines": {
"node": "14 || >=16.14"
}
},
"node_modules/path-scurry/node_modules/minipass": {
"version": "7.0.4",
"resolved": "https://registry.npmjs.org/minipass/-/minipass-7.0.4.tgz",
"integrity": "sha512-jYofLM5Dam9279rdkWzqHozUo4ybjdZmCsDHePy5V/PbBcVMiSZR97gmAy45aqi8CK1lG2ECd356FU86avfwUQ==",
"dev": true,
"engines": {
"node": ">=16 || 14 >=14.17"
}
},
"node_modules/path-to-regexp": {
"version": "6.2.1",
"resolved": "https://registry.npmjs.org/path-to-regexp/-/path-to-regexp-6.2.1.tgz",
@@ -10913,6 +11495,12 @@
"resolved": "https://registry.npmjs.org/react-is/-/react-is-16.13.1.tgz",
"integrity": "sha512-24e6ynE2H+OKt4kqsOvNd8kBpV65zoxbA4BVsEOB3ARVWQki/DHzaUoC5KuON/BiccDaCCTZBuOcfZs70kR8bQ=="
},
"node_modules/proto-list": {
"version": "1.2.4",
"resolved": "https://registry.npmjs.org/proto-list/-/proto-list-1.2.4.tgz",
"integrity": "sha512-vtK/94akxsTMhe0/cbfpR+syPuszcuwhqVjJq26CuNDgFGj682oRBXOP5MJpv2r7JtE8MsiepGIqvvOTBwn2vA==",
"dev": true
},
"node_modules/proxy-from-env": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/proxy-from-env/-/proxy-from-env-1.1.0.tgz",
@@ -11715,6 +12303,74 @@
}
}
},
"node_modules/sequelize-cli": {
"version": "6.6.2",
"resolved": "https://registry.npmjs.org/sequelize-cli/-/sequelize-cli-6.6.2.tgz",
"integrity": "sha512-V8Oh+XMz2+uquLZltZES6MVAD+yEnmMfwfn+gpXcDiwE3jyQygLt4xoI0zG8gKt6cRcs84hsKnXAKDQjG/JAgg==",
"dev": true,
"dependencies": {
"cli-color": "^2.0.3",
"fs-extra": "^9.1.0",
"js-beautify": "^1.14.5",
"lodash": "^4.17.21",
"resolve": "^1.22.1",
"umzug": "^2.3.0",
"yargs": "^16.2.0"
},
"bin": {
"sequelize": "lib/sequelize",
"sequelize-cli": "lib/sequelize"
},
"engines": {
"node": ">=10.0.0"
}
},
"node_modules/sequelize-cli/node_modules/cliui": {
"version": "7.0.4",
"resolved": "https://registry.npmjs.org/cliui/-/cliui-7.0.4.tgz",
"integrity": "sha512-OcRE68cOsVMXp1Yvonl/fzkQOyjLSu/8bhPDfQt0e0/Eb283TKP20Fs2MqoPsr9SwA595rRCA+QMzYc9nBP+JQ==",
"dev": true,
"dependencies": {
"string-width": "^4.2.0",
"strip-ansi": "^6.0.0",
"wrap-ansi": "^7.0.0"
}
},
"node_modules/sequelize-cli/node_modules/wrap-ansi": {
"version": "7.0.0",
"resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-7.0.0.tgz",
"integrity": "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==",
"dev": true,
"dependencies": {
"ansi-styles": "^4.0.0",
"string-width": "^4.1.0",
"strip-ansi": "^6.0.0"
},
"engines": {
"node": ">=10"
},
"funding": {
"url": "https://github.com/chalk/wrap-ansi?sponsor=1"
}
},
"node_modules/sequelize-cli/node_modules/yargs": {
"version": "16.2.0",
"resolved": "https://registry.npmjs.org/yargs/-/yargs-16.2.0.tgz",
"integrity": "sha512-D1mvvtDG0L5ft/jGWkLpG1+m0eQxOfaBvTNELraWj22wSVUMWxZUvYgJYcKh6jGGIkJFhH4IZPQhR4TKpc8mBw==",
"dev": true,
"dependencies": {
"cliui": "^7.0.2",
"escalade": "^3.1.1",
"get-caller-file": "^2.0.5",
"require-directory": "^2.1.1",
"string-width": "^4.2.0",
"y18n": "^5.0.5",
"yargs-parser": "^20.2.2"
},
"engines": {
"node": ">=10"
}
},
"node_modules/sequelize-pool": {
"version": "7.1.0",
"resolved": "https://registry.npmjs.org/sequelize-pool/-/sequelize-pool-7.1.0.tgz",
@@ -11724,9 +12380,9 @@
}
},
"node_modules/sequelize-typescript": {
"version": "2.1.5",
"resolved": "https://registry.npmjs.org/sequelize-typescript/-/sequelize-typescript-2.1.5.tgz",
"integrity": "sha512-x1CNODct8gJyfZPwEZBU5uVGNwgJI2Fda913ZxD5ZtCSRyTDPBTS/0uXciF+MlCpyqjpmoCAPtudQWzw579bzA==",
"version": "2.1.6",
"resolved": "https://registry.npmjs.org/sequelize-typescript/-/sequelize-typescript-2.1.6.tgz",
"integrity": "sha512-Vc2N++3en346RsbGjL3h7tgAl2Y7V+2liYTAOZ8XL0KTw3ahFHsyAUzOwct51n+g70I1TOUDgs06Oh6+XGcFkQ==",
"dependencies": {
"glob": "7.2.0"
},
@@ -12361,6 +13017,27 @@
"node": ">=8"
}
},
"node_modules/string-width-cjs": {
"name": "string-width",
"version": "4.2.3",
"resolved": "https://registry.npmjs.org/string-width/-/string-width-4.2.3.tgz",
"integrity": "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g==",
"dev": true,
"dependencies": {
"emoji-regex": "^8.0.0",
"is-fullwidth-code-point": "^3.0.0",
"strip-ansi": "^6.0.1"
},
"engines": {
"node": ">=8"
}
},
"node_modules/string-width-cjs/node_modules/emoji-regex": {
"version": "8.0.0",
"resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz",
"integrity": "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A==",
"dev": true
},
"node_modules/string-width/node_modules/emoji-regex": {
"version": "8.0.0",
"resolved": "https://registry.npmjs.org/emoji-regex/-/emoji-regex-8.0.0.tgz",
@@ -12449,6 +13126,19 @@
"node": ">=8"
}
},
"node_modules/strip-ansi-cjs": {
"name": "strip-ansi",
"version": "6.0.1",
"resolved": "https://registry.npmjs.org/strip-ansi/-/strip-ansi-6.0.1.tgz",
"integrity": "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A==",
"dev": true,
"dependencies": {
"ansi-regex": "^5.0.1"
},
"engines": {
"node": ">=8"
}
},
"node_modules/strip-bom": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/strip-bom/-/strip-bom-4.0.0.tgz",
@@ -13093,6 +13783,16 @@
"readable-stream": "3"
}
},
"node_modules/timers-ext": {
"version": "0.1.7",
"resolved": "https://registry.npmjs.org/timers-ext/-/timers-ext-0.1.7.tgz",
"integrity": "sha512-b85NUNzTSdodShTIbky6ZF02e8STtVVfD+fu4aXXShEELpozH+bCpJLYMPZbsABN2wDH7fJpqIoXxJpzbf0NqQ==",
"dev": true,
"dependencies": {
"es5-ext": "~0.10.46",
"next-tick": "1"
}
},
"node_modules/tmp": {
"version": "0.0.33",
"resolved": "https://registry.npmjs.org/tmp/-/tmp-0.0.33.tgz",
@@ -13252,6 +13952,12 @@
"integrity": "sha512-Xni35NKzjgMrwevysHTCArtLDpPvye8zV/0E4EyYn43P7/7qvQwPh9BGkHewbMulVntbigmcT7rdX3BNo9wRJg==",
"dev": true
},
"node_modules/type": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/type/-/type-1.2.0.tgz",
"integrity": "sha512-+5nt5AAniqsCnu2cEQQdpzCAh33kVx8n0VoFidKpB1dVVLAN/F+bgVOqOJqOnEnrhp222clB5p3vUlD+1QAnfg==",
"dev": true
},
"node_modules/type-check": {
"version": "0.4.0",
"resolved": "https://registry.npmjs.org/type-check/-/type-check-0.4.0.tgz",
@@ -13382,6 +14088,18 @@
"node": ">=0.8.0"
}
},
"node_modules/umzug": {
"version": "2.3.0",
"resolved": "https://registry.npmjs.org/umzug/-/umzug-2.3.0.tgz",
"integrity": "sha512-Z274K+e8goZK8QJxmbRPhl89HPO1K+ORFtm6rySPhFKfKc5GHhqdzD0SGhSWHkzoXasqJuItdhorSvY7/Cgflw==",
"dev": true,
"dependencies": {
"bluebird": "^3.7.2"
},
"engines": {
"node": ">=6.0.0"
}
},
"node_modules/unbox-primitive": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/unbox-primitive/-/unbox-primitive-1.0.2.tgz",
@@ -13788,6 +14506,24 @@
"node": ">=8"
}
},
"node_modules/wrap-ansi-cjs": {
"name": "wrap-ansi",
"version": "7.0.0",
"resolved": "https://registry.npmjs.org/wrap-ansi/-/wrap-ansi-7.0.0.tgz",
"integrity": "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q==",
"dev": true,
"dependencies": {
"ansi-styles": "^4.0.0",
"string-width": "^4.1.0",
"strip-ansi": "^6.0.0"
},
"engines": {
"node": ">=10"
},
"funding": {
"url": "https://github.com/chalk/wrap-ansi?sponsor=1"
}
},
"node_modules/wrappy": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz",

View File

@@ -1,18 +1,21 @@
{
"name": "serpbear",
"version": "0.3.4",
"version": "1.0.0",
"private": true,
"scripts": {
"dev": "next dev",
"build": "next build",
"start": "next start",
"cron": "node cron.js",
"prestart": "npm run db:migrate",
"start:all": "concurrently npm:start npm:cron",
"lint": "next lint",
"lint:css": "stylelint styles/*.css",
"test": "jest --watch --verbose",
"test:ci": "jest --ci",
"test:cv": "jest --coverage --coverageDirectory='coverage'",
"db:migrate": "sequelize-cli db:migrate --env production",
"db:revert": "sequelize-cli db:migrate:undo --env production",
"release": "standard-version"
},
"dependencies": {
@@ -33,7 +36,7 @@
"jsonwebtoken": "^9.0.2",
"msw": "^0.49.0",
"next": "^12.3.4",
"nodemailer": "^6.8.0",
"nodemailer": "^6.9.9",
"react": "18.2.0",
"react-chartjs-2": "^4.3.1",
"react-dom": "18.2.0",
@@ -44,7 +47,7 @@
"react-window": "^1.8.8",
"reflect-metadata": "^0.1.13",
"sequelize": "^6.34.0",
"sequelize-typescript": "^2.1.5",
"sequelize-typescript": "^2.1.6",
"sqlite3": "^5.1.6"
},
"devDependencies": {
@@ -73,6 +76,7 @@
"prettier": "^2.7.1",
"resize-observer-polyfill": "^1.5.1",
"sass": "^1.55.0",
"sequelize-cli": "^6.6.2",
"standard-version": "^9.5.0",
"stylelint-config-standard": "^29.0.0",
"tailwindcss": "^3.1.8",

48
pages/api/domain.ts Normal file
View File

@@ -0,0 +1,48 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import Cryptr from 'cryptr';
import db from '../../database/database';
import Domain from '../../database/models/domain';
import verifyUser from '../../utils/verifyUser';
type DomainGetResponse = {
domain?: DomainType | null
error?: string|null,
}
export default async function handler(req: NextApiRequest, res: NextApiResponse) {
const authorized = verifyUser(req, res);
if (authorized === 'authorized' && req.method === 'GET') {
await db.sync();
return getDomain(req, res);
}
return res.status(401).json({ error: authorized });
}
const getDomain = async (req: NextApiRequest, res: NextApiResponse<DomainGetResponse>) => {
if (!req.query.domain && typeof req.query.domain !== 'string') {
return res.status(400).json({ error: 'Domain Name is Required!' });
}
try {
const query = { domain: req.query.domain as string };
const foundDomain:Domain| null = await Domain.findOne({ where: query });
const parsedDomain = foundDomain?.get({ plain: true }) || false;
if (parsedDomain && parsedDomain.search_console) {
try {
const cryptr = new Cryptr(process.env.SECRET as string);
const scData = JSON.parse(parsedDomain.search_console);
scData.client_email = scData.client_email ? cryptr.decrypt(scData.client_email) : '';
scData.private_key = scData.private_key ? cryptr.decrypt(scData.private_key) : '';
parsedDomain.search_console = JSON.stringify(scData);
} catch (error) {
console.log('[Error] Parsing Search Console Keys.');
}
}
return res.status(200).json({ domain: parsedDomain });
} catch (error) {
console.log('[ERROR] Getting Domain: ', error);
return res.status(400).json({ error: 'Error Loading Domain' });
}
};

View File

@@ -1,10 +1,11 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import Cryptr from 'cryptr';
import db from '../../database/database';
import Domain from '../../database/models/domain';
import Keyword from '../../database/models/keyword';
import getdomainStats from '../../utils/domains';
import verifyUser from '../../utils/verifyUser';
import { removeLocalSCData } from '../../utils/searchConsole';
import { checkSerchConsoleIntegration, removeLocalSCData } from '../../utils/searchConsole';
type DomainsGetRes = {
domains: DomainType[]
@@ -53,7 +54,13 @@ export const getDomains = async (req: NextApiRequest, res: NextApiResponse<Domai
const withStats = !!req?.query?.withstats;
try {
const allDomains: Domain[] = await Domain.findAll();
const formattedDomains: DomainType[] = allDomains.map((el) => el.get({ plain: true }));
const formattedDomains: DomainType[] = allDomains.map((el) => {
const domainItem:DomainType = el.get({ plain: true });
const scData = domainItem?.search_console ? JSON.parse(domainItem.search_console) : {};
const { client_email, private_key } = scData;
const searchConsoleData = scData ? { ...scData, client_email: client_email ? 'true' : '', private_key: private_key ? 'true' : '' } : {};
return { ...domainItem, search_console: JSON.stringify(searchConsoleData) };
});
const theDomains: DomainType[] = withStats ? await getdomainStats(formattedDomains) : allDomains;
return res.status(200).json({ domains: theDomains });
} catch (error) {
@@ -69,7 +76,7 @@ const addDomain = async (req: NextApiRequest, res: NextApiResponse<DomainsAddRes
domains.forEach((domain: string) => {
domainsToAdd.push({
domain: domain.trim(),
slug: domain.trim().replaceAll('-', '_').replaceAll('.', '-'),
slug: domain.trim().replaceAll('-', '_').replaceAll('.', '-').replaceAll('/', '-'),
lastUpdated: new Date().toJSON(),
added: new Date().toJSON(),
});
@@ -108,17 +115,28 @@ export const updateDomain = async (req: NextApiRequest, res: NextApiResponse<Dom
return res.status(400).json({ domain: null, error: 'Domain is Required!' });
}
const { domain } = req.query || {};
const { notification_interval, notification_emails } = req.body;
const { notification_interval, notification_emails, search_console } = req.body as DomainSettings;
try {
const domainToUpdate: Domain|null = await Domain.findOne({ where: { domain } });
// Validate Search Console API Data
if (domainToUpdate && search_console?.client_email && search_console?.private_key) {
const theDomainObj = domainToUpdate.get({ plain: true });
const isSearchConsoleAPIValid = await checkSerchConsoleIntegration({ ...theDomainObj, search_console: JSON.stringify(search_console) });
if (!isSearchConsoleAPIValid.isValid) {
return res.status(400).json({ domain: null, error: isSearchConsoleAPIValid.error });
}
const cryptr = new Cryptr(process.env.SECRET as string);
search_console.client_email = search_console.client_email ? cryptr.encrypt(search_console.client_email.trim()) : '';
search_console.private_key = search_console.private_key ? cryptr.encrypt(search_console.private_key.trim()) : '';
}
if (domainToUpdate) {
domainToUpdate.set({ notification_interval, notification_emails });
domainToUpdate.set({ notification_interval, notification_emails, search_console: JSON.stringify(search_console) });
await domainToUpdate.save();
}
return res.status(200).json({ domain: domainToUpdate });
} catch (error) {
console.log('[ERROR] Updating Domain: ', req.query.domain, error);
return res.status(400).json({ domain: null, error: 'Error Updating Domain' });
return res.status(400).json({ domain: null, error: 'Error Updating Domain. An Unknown Error Occured.' });
}
};

View File

@@ -1,8 +1,9 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import db from '../../database/database';
import { getCountryInsight, getKeywordsInsight, getPagesInsight } from '../../utils/insight';
import { fetchDomainSCData, readLocalSCData } from '../../utils/searchConsole';
import { fetchDomainSCData, getSearchConsoleApiInfo, readLocalSCData } from '../../utils/searchConsole';
import verifyUser from '../../utils/verifyUser';
import Domain from '../../database/models/domain';
type SCInsightRes = {
data: InsightDataType | null,
@@ -23,9 +24,6 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse)
const getDomainSearchConsoleInsight = async (req: NextApiRequest, res: NextApiResponse<SCInsightRes>) => {
if (!req.query.domain && typeof req.query.domain !== 'string') return res.status(400).json({ data: null, error: 'Domain is Missing.' });
if (!!(process.env.SEARCH_CONSOLE_PRIVATE_KEY && process.env.SEARCH_CONSOLE_CLIENT_EMAIL) === false) {
return res.status(200).json({ data: null, error: 'Google Search Console Not Integrated' });
}
const domainname = (req.query.domain as string).replaceAll('-', '.').replaceAll('_', '-');
const getInsightFromSCData = (localSCData: SCDomainDataType): InsightDataType => {
const { stats = [] } = localSCData;
@@ -37,17 +35,26 @@ const getDomainSearchConsoleInsight = async (req: NextApiRequest, res: NextApiRe
// First try and read the Local SC Domain Data file.
const localSCData = await readLocalSCData(domainname);
const oldFetchedDate = localSCData.lastFetched;
const fetchTimeDiff = new Date().getTime() - (oldFetchedDate ? new Date(oldFetchedDate as string).getTime() : 0);
if (localSCData && localSCData.stats && localSCData.stats.length && fetchTimeDiff <= 86400000) {
const response = getInsightFromSCData(localSCData);
return res.status(200).json({ data: response });
if (localSCData) {
const oldFetchedDate = localSCData.lastFetched;
const fetchTimeDiff = new Date().getTime() - (oldFetchedDate ? new Date(oldFetchedDate as string).getTime() : 0);
if (localSCData.stats && localSCData.stats.length && fetchTimeDiff <= 86400000) {
const response = getInsightFromSCData(localSCData);
return res.status(200).json({ data: response });
}
}
// If the Local SC Domain Data file does not exist, fetch from Googel Search Console.
try {
const scData = await fetchDomainSCData(domainname);
const query = { domain: domainname };
const foundDomain:Domain| null = await Domain.findOne({ where: query });
const domainObj: DomainType = foundDomain && foundDomain.get({ plain: true });
const scDomainAPI = await getSearchConsoleApiInfo(domainObj);
if (!(scDomainAPI.client_email && scDomainAPI.private_key)) {
return res.status(200).json({ data: null, error: 'Google Search Console is not Integrated.' });
}
const scData = await fetchDomainSCData(domainObj, scDomainAPI);
const response = getInsightFromSCData(scData);
return res.status(200).json({ data: response });
} catch (error) {

View File

@@ -45,7 +45,7 @@ const getKeywords = async (req: NextApiRequest, res: NextApiResponse<KeywordsGet
if (!req.query.domain && typeof req.query.domain !== 'string') {
return res.status(400).json({ error: 'Domain is Required!' });
}
const domain = (req.query.domain as string).replaceAll('-', '.').replaceAll('_', '-');
const domain = (req.query.domain as string);
const integratedSC = process.env.SEARCH_CONSOLE_PRIVATE_KEY && process.env.SEARCH_CONSOLE_CLIENT_EMAIL;
const domainSCData = integratedSC ? await readLocalSCData(domain) : false;
@@ -79,13 +79,14 @@ const addKeywords = async (req: NextApiRequest, res: NextApiResponse<KeywordsGet
const keywordsToAdd: any = []; // QuickFIX for bug: https://github.com/sequelize/sequelize-typescript/issues/936
keywords.forEach((kwrd: KeywordAddPayload) => {
const { keyword, device, country, domain, tags } = kwrd;
const { keyword, device, country, domain, tags, city } = kwrd;
const tagsArray = tags ? tags.split(',').map((item:string) => item.trim()) : [];
const newKeyword = {
keyword,
device,
domain,
country,
city,
position: 0,
updating: true,
history: JSON.stringify({}),

View File

@@ -1,7 +1,7 @@
import type { NextApiRequest, NextApiResponse } from 'next';
import db from '../../database/database';
import Domain from '../../database/models/domain';
import { fetchDomainSCData, readLocalSCData } from '../../utils/searchConsole';
import { fetchDomainSCData, getSearchConsoleApiInfo, readLocalSCData } from '../../utils/searchConsole';
import verifyUser from '../../utils/verifyUser';
type searchConsoleRes = {
@@ -31,18 +31,20 @@ export default async function handler(req: NextApiRequest, res: NextApiResponse)
const getDomainSearchConsoleData = async (req: NextApiRequest, res: NextApiResponse<searchConsoleRes>) => {
if (!req.query.domain && typeof req.query.domain !== 'string') return res.status(400).json({ data: null, error: 'Domain is Missing.' });
if (!!(process.env.SEARCH_CONSOLE_PRIVATE_KEY && process.env.SEARCH_CONSOLE_CLIENT_EMAIL) === false) {
return res.status(200).json({ data: null, error: 'Google Search Console Not Integrated' });
}
const domainname = (req.query.domain as string).replaceAll('-', '.').replaceAll('_', '-');
const localSCData = await readLocalSCData(domainname);
console.log(localSCData && localSCData.thirtyDays && localSCData.thirtyDays.length);
if (localSCData && localSCData.thirtyDays && localSCData.thirtyDays.length) {
return res.status(200).json({ data: localSCData });
}
try {
const scData = await fetchDomainSCData(domainname);
const query = { domain: domainname };
const foundDomain:Domain| null = await Domain.findOne({ where: query });
const domainObj: DomainType = foundDomain && foundDomain.get({ plain: true });
const scDomainAPI = await getSearchConsoleApiInfo(domainObj);
if (!(scDomainAPI.client_email && scDomainAPI.private_key)) {
return res.status(200).json({ data: null, error: 'Google Search Console is not Integrated.' });
}
const scData = await fetchDomainSCData(domainObj, scDomainAPI);
return res.status(200).json({ data: scData });
} catch (error) {
console.log('[ERROR] Getting Search Console Data for: ', domainname, error);
@@ -53,9 +55,9 @@ const getDomainSearchConsoleData = async (req: NextApiRequest, res: NextApiRespo
const cronRefreshSearchConsoleData = async (req: NextApiRequest, res: NextApiResponse<searchConsoleCRONRes>) => {
try {
const allDomainsRaw = await Domain.findAll();
const Domains: Domain[] = allDomainsRaw.map((el) => el.get({ plain: true }));
const Domains: DomainType[] = allDomainsRaw.map((el) => el.get({ plain: true }));
for (const domain of Domains) {
await fetchDomainSCData(domain.domain);
await fetchDomainSCData(domain);
}
return res.status(200).json({ status: 'completed' });
} catch (error) {

View File

@@ -42,9 +42,11 @@ const updateSettings = async (req: NextApiRequest, res: NextApiResponse<Settings
}
try {
const cryptr = new Cryptr(process.env.SECRET as string);
const scaping_api = settings.scaping_api ? cryptr.encrypt(settings.scaping_api) : '';
const smtp_password = settings.smtp_password ? cryptr.encrypt(settings.smtp_password) : '';
const securedSettings = { ...settings, scaping_api, smtp_password };
const scaping_api = settings.scaping_api ? cryptr.encrypt(settings.scaping_api.trim()) : '';
const smtp_password = settings.smtp_password ? cryptr.encrypt(settings.smtp_password.trim()) : '';
const search_console_client_email = settings.search_console_client_email ? cryptr.encrypt(settings.search_console_client_email.trim()) : '';
const search_console_private_key = settings.search_console_private_key ? cryptr.encrypt(settings.search_console_private_key.trim()) : '';
const securedSettings = { ...settings, scaping_api, smtp_password, search_console_client_email, search_console_private_key };
await writeFile(`${process.cwd()}/data/settings.json`, JSON.stringify(securedSettings), { encoding: 'utf-8' });
return res.status(200).json({ settings });
@@ -67,12 +69,17 @@ export const getAppSettings = async () : Promise<SettingsType> => {
const cryptr = new Cryptr(process.env.SECRET as string);
const scaping_api = settings.scaping_api ? cryptr.decrypt(settings.scaping_api) : '';
const smtp_password = settings.smtp_password ? cryptr.decrypt(settings.smtp_password) : '';
const search_console_client_email = settings.search_console_client_email ? cryptr.decrypt(settings.search_console_client_email) : '';
const search_console_private_key = settings.search_console_private_key ? cryptr.decrypt(settings.search_console_private_key) : '';
decryptedSettings = {
...settings,
scaping_api,
smtp_password,
search_console_integrated: !!(process.env.SEARCH_CONSOLE_PRIVATE_KEY && process.env.SEARCH_CONSOLE_CLIENT_EMAIL),
available_scapers: allScrapers.map((scraper) => ({ label: scraper.name, value: scraper.id })),
search_console_client_email,
search_console_private_key,
search_console_integrated: !!(process.env.SEARCH_CONSOLE_PRIVATE_KEY && process.env.SEARCH_CONSOLE_CLIENT_EMAIL)
|| !!(search_console_client_email && search_console_private_key),
available_scapers: allScrapers.map((scraper) => ({ label: scraper.name, value: scraper.id, allowsCity: !!scraper.allowsCity })),
failed_queue: failedQueue,
screenshot_key: screenshotAPIKey,
};
@@ -94,6 +101,9 @@ export const getAppSettings = async () : Promise<SettingsType> => {
smtp_password: '',
scrape_retry: false,
screenshot_key: screenshotAPIKey,
search_console: true,
search_console_client_email: '',
search_console_private_key: '',
};
const otherSettings = {
available_scapers: allScrapers.map((scraper) => ({ label: scraper.name, value: scraper.id })),

View File

@@ -1,9 +1,7 @@
import React, { useEffect, useMemo, useState } from 'react';
import React, { useMemo, useState } from 'react';
import type { NextPage } from 'next';
import Head from 'next/head';
import { useRouter } from 'next/router';
// import { useQuery } from 'react-query';
// import toast from 'react-hot-toast';
import { CSSTransition } from 'react-transition-group';
import Sidebar from '../../../components/common/Sidebar';
import TopBar from '../../../components/common/TopBar';
@@ -16,21 +14,20 @@ import Settings from '../../../components/settings/Settings';
import { useFetchDomains } from '../../../services/domains';
import { useFetchKeywords } from '../../../services/keywords';
import { useFetchSettings } from '../../../services/settings';
import AddKeywords from '../../../components/keywords/AddKeywords';
const SingleDomain: NextPage = () => {
const router = useRouter();
const [noScrapprtError, setNoScrapprtError] = useState(false);
const [showAddKeywords, setShowAddKeywords] = useState(false);
const [showAddDomain, setShowAddDomain] = useState(false);
const [showDomainSettings, setShowDomainSettings] = useState(false);
const [showSettings, setShowSettings] = useState(false);
const [keywordSPollInterval, setKeywordSPollInterval] = useState<undefined|number>(undefined);
const { data: appSettings } = useFetchSettings();
const { data: appSettingsData, isLoading: isAppSettingsLoading } = useFetchSettings();
const { data: domainsData } = useFetchDomains(router);
const { keywordsData, keywordsLoading } = useFetchKeywords(router, setKeywordSPollInterval, keywordSPollInterval);
const theDomains: DomainType[] = (domainsData && domainsData.domains) || [];
const theKeywords: KeywordType[] = keywordsData && keywordsData.keywords;
const appSettings: SettingsType = appSettingsData?.settings || {};
const { scraper_type = '', available_scapers = [] } = appSettings;
const activeScraper = useMemo(() => available_scapers.find((scraper) => scraper.value === scraper_type), [scraper_type, available_scapers]);
const activDomain: DomainType|null = useMemo(() => {
let active:DomainType|null = null;
@@ -40,18 +37,18 @@ const SingleDomain: NextPage = () => {
return active;
}, [router.query.slug, domainsData]);
useEffect(() => {
// console.log('appSettings.settings: ', appSettings && appSettings.settings);
if (appSettings && appSettings.settings && (!appSettings.settings.scraper_type || (appSettings.settings.scraper_type === 'none'))) {
setNoScrapprtError(true);
}
}, [appSettings]);
const domainHasScAPI = useMemo(() => {
const doaminSc = activDomain?.search_console ? JSON.parse(activDomain.search_console) : {};
return !!(doaminSc?.client_email && doaminSc?.private_key);
}, [activDomain]);
// console.log('Domains Data:', router, activDomain, theKeywords);
const { keywordsData, keywordsLoading } = useFetchKeywords(router, activDomain?.domain || '', setKeywordSPollInterval, keywordSPollInterval);
const theDomains: DomainType[] = (domainsData && domainsData.domains) || [];
const theKeywords: KeywordType[] = keywordsData && keywordsData.keywords;
return (
<div className="Domain ">
{noScrapprtError && (
{((!scraper_type || (scraper_type === 'none')) && !isAppSettingsLoading) && (
<div className=' p-3 bg-red-600 text-white text-sm text-center'>
A Scrapper/Proxy has not been set up Yet. Open Settings to set it up and start using the app.
</div>
@@ -80,7 +77,7 @@ const SingleDomain: NextPage = () => {
keywords={theKeywords}
showAddModal={showAddKeywords}
setShowAddModal={setShowAddKeywords}
isConsoleIntegrated={!!(appSettings && appSettings?.settings?.search_console_integrated) }
isConsoleIntegrated={!!(appSettings && appSettings.search_console_integrated) || domainHasScAPI }
/>
</div>
</div>
@@ -98,6 +95,15 @@ const SingleDomain: NextPage = () => {
<CSSTransition in={showSettings} timeout={300} classNames="settings_anim" unmountOnExit mountOnEnter>
<Settings closeSettings={() => setShowSettings(false)} />
</CSSTransition>
<CSSTransition in={showAddKeywords} timeout={300} classNames="modal_anim" unmountOnExit mountOnEnter>
<AddKeywords
domain={activDomain?.domain || ''}
scraperName={activeScraper?.label || ''}
keywords={theKeywords}
allowsCity={!!activeScraper?.allowsCity}
closeModal={() => setShowAddKeywords(false)}
/>
</CSSTransition>
</div>
);
};

View File

@@ -39,6 +39,11 @@ const DiscoverPage: NextPage = () => {
return active;
}, [router.query.slug, domainsData]);
const domainHasScAPI = useMemo(() => {
const doaminSc = activDomain?.search_console ? JSON.parse(activDomain.search_console) : {};
return !!(doaminSc?.client_email && doaminSc?.private_key);
}, [activDomain]);
return (
<div className="Domain ">
{activDomain && activDomain.domain
@@ -65,7 +70,7 @@ const DiscoverPage: NextPage = () => {
isLoading={keywordsLoading || isFetching}
domain={activDomain}
keywords={theKeywords}
isConsoleIntegrated={scConnected}
isConsoleIntegrated={scConnected || domainHasScAPI}
/>
</div>
</div>

View File

@@ -39,6 +39,11 @@ const InsightPage: NextPage = () => {
return active;
}, [router.query.slug, domainsData]);
const domainHasScAPI = useMemo(() => {
const doaminSc = activDomain?.search_console ? JSON.parse(activDomain.search_console) : {};
return !!(doaminSc?.client_email && doaminSc?.private_key);
}, [activDomain]);
return (
<div className="Domain ">
{activDomain && activDomain.domain
@@ -65,7 +70,7 @@ const InsightPage: NextPage = () => {
isLoading={false}
domain={activDomain}
insight={theInsight}
isConsoleIntegrated={scConnected}
isConsoleIntegrated={scConnected || domainHasScAPI}
/>
</div>
</div>

View File

@@ -16,12 +16,14 @@ type thumbImages = { [domain:string] : string }
const Domains: NextPage = () => {
const router = useRouter();
const [noScrapprtError, setNoScrapprtError] = useState(false);
// const [noScrapprtError, setNoScrapprtError] = useState(false);
const [showSettings, setShowSettings] = useState(false);
const [showAddDomain, setShowAddDomain] = useState(false);
const [domainThumbs, setDomainThumbs] = useState<thumbImages>({});
const { data: appSettings } = useFetchSettings();
const { data: appSettingsData, isLoading: isAppSettingsLoading } = useFetchSettings();
const { data: domainsData, isLoading } = useFetchDomains(router, true);
const appSettings:SettingsType = appSettingsData?.settings || {};
const { scraper_type = '' } = appSettings;
const totalKeywords = useMemo(() => {
let keywords = 0;
@@ -33,29 +35,33 @@ const Domains: NextPage = () => {
return keywords;
}, [domainsData]);
const domainSCAPiObj = useMemo(() => {
const domainsSCAPI:{ [ID:string] : boolean } = {};
if (domainsData?.domains) {
domainsData.domains.forEach(async (domain:DomainType) => {
const doaminSc = domain?.search_console ? JSON.parse(domain.search_console) : {};
domainsSCAPI[domain.ID] = doaminSc.client_email && doaminSc.private_key;
});
}
return domainsSCAPI;
}, [domainsData]);
useEffect(() => {
if (domainsData?.domains && domainsData.domains.length > 0 && appSettings?.settings?.screenshot_key) {
if (domainsData?.domains && domainsData.domains.length > 0 && appSettings.screenshot_key) {
domainsData.domains.forEach(async (domain:DomainType) => {
if (domain.domain) {
const domainThumb = await fetchDomainScreenshot(domain.domain, appSettings.settings.screenshot_key);
const domainThumb = await fetchDomainScreenshot(domain.domain, appSettings.screenshot_key || '');
if (domainThumb) {
setDomainThumbs((currentThumbs) => ({ ...currentThumbs, [domain.domain]: domainThumb }));
}
}
});
}
}, [domainsData, appSettings]);
useEffect(() => {
// console.log('appSettings.settings: ', appSettings && appSettings.settings);
if (appSettings && appSettings.settings && (!appSettings.settings.scraper_type || (appSettings.settings.scraper_type === 'none'))) {
setNoScrapprtError(true);
}
}, [appSettings]);
}, [domainsData, appSettings.screenshot_key]);
const manuallyUpdateThumb = async (domain: string) => {
if (domain && appSettings?.settings?.screenshot_key) {
const domainThumb = await fetchDomainScreenshot(domain, appSettings.settings.screenshot_key, true);
if (domain && appSettings.screenshot_key) {
const domainThumb = await fetchDomainScreenshot(domain, appSettings.screenshot_key, true);
if (domainThumb) {
toast(`${domain} Screenshot Updated Successfully!`, { icon: '✔️' });
setDomainThumbs((currentThumbs) => ({ ...currentThumbs, [domain]: domainThumb }));
@@ -67,7 +73,7 @@ const Domains: NextPage = () => {
return (
<div data-testid="domains" className="Domain flex flex-col min-h-screen">
{noScrapprtError && (
{((!scraper_type || (scraper_type === 'none')) && !isAppSettingsLoading) && (
<div className=' p-3 bg-red-600 text-white text-sm text-center'>
A Scrapper/Proxy has not been set up Yet. Open Settings to set it up and start using the app.
</div>
@@ -99,7 +105,7 @@ const Domains: NextPage = () => {
key={domain.ID}
domain={domain}
selected={false}
isConsoleIntegrated={!!(appSettings && appSettings?.settings?.search_console_integrated) }
isConsoleIntegrated={!!(appSettings && appSettings.search_console_integrated) || !!domainSCAPiObj[domain.ID] }
thumb={domainThumbs[domain.domain]}
updateThumb={manuallyUpdateThumb}
// isConsoleIntegrated={false}
@@ -125,7 +131,7 @@ const Domains: NextPage = () => {
<Settings closeSettings={() => setShowSettings(false)} />
</CSSTransition>
<footer className='text-center flex flex-1 justify-center pb-5 items-end'>
<span className='text-gray-500 text-xs'><a href='https://github.com/towfiqi/serpbear' target="_blank" rel='noreferrer'>SerpBear v{appSettings?.settings?.version || '0.0.0'}</a></span>
<span className='text-gray-500 text-xs'><a href='https://github.com/towfiqi/serpbear' target="_blank" rel='noreferrer'>SerpBear v{appSettings.version || '0.0.0'}</a></span>
</footer>
<Toaster position='bottom-center' containerClassName="react_toaster" />
</div>

View File

@@ -5,6 +5,8 @@ import serply from './services/serply';
import spaceserp from './services/spaceserp';
import proxy from './services/proxy';
import searchapi from './services/searchapi';
import valueSerp from './services/valueserp';
import serper from './services/serper';
export default [
scrapingRobot,
@@ -14,4 +16,6 @@ export default [
spaceserp,
proxy,
searchapi,
valueSerp,
serper,
];

View File

@@ -1,7 +1,16 @@
import countries from '../../utils/countries';
interface SearchApiResult {
title: string,
link: string,
position: number,
}
const searchapi:ScraperSettings = {
id: 'searchapi',
name: 'SearchApi.io',
website: 'searchapi.io',
allowsCity: true,
headers: (keyword, settings) => {
return {
'Content-Type': 'application/json',
@@ -9,7 +18,10 @@ const searchapi:ScraperSettings = {
};
},
scrapeURL: (keyword) => {
return `https://www.searchapi.io/api/v1/search?engine=google&q=${encodeURI(keyword.keyword)}&num=100&gl=${keyword.country}&device=${keyword.device}`;
const country = keyword.country || 'US';
const countryName = countries[country][0];
const location = keyword.city && countryName ? `&location=${encodeURI(`${keyword.city},${countryName}`)}` : '';
return `https://www.searchapi.io/api/v1/search?engine=google&q=${encodeURI(keyword.keyword)}&num=100&gl=${country}&device=${keyword.device}${location}`;
},
resultObjectKey: 'organic_results',
serpExtractor: (content) => {
@@ -29,10 +41,4 @@ const searchapi:ScraperSettings = {
},
};
interface SearchApiResult {
title: string,
link: string,
position: number,
}
export default searchapi;

View File

@@ -1,3 +1,5 @@
import countries from '../../utils/countries';
interface SerpApiResult {
title: string,
link: string,
@@ -8,6 +10,7 @@ const serpapi:ScraperSettings = {
id: 'serpapi',
name: 'SerpApi.com',
website: 'serpapi.com',
allowsCity: true,
headers: (keyword, settings) => {
return {
'Content-Type': 'application/json',
@@ -15,7 +18,9 @@ const serpapi:ScraperSettings = {
};
},
scrapeURL: (keyword, settings) => {
return `https://serpapi.com/search?q=${encodeURI(keyword.keyword)}&num=100&gl=${keyword.country}&device=${keyword.device}&api_key=${settings.scaping_api}`;
const countryName = countries[keyword.country || 'US'][0];
const location = keyword.city && keyword.country ? `&location=${encodeURI(`${keyword.city},${countryName}`)}` : '';
return `https://serpapi.com/search?q=${encodeURI(keyword.keyword)}&num=100&gl=${keyword.country}&device=${keyword.device}${location}&api_key=${settings.scaping_api}`;
},
resultObjectKey: 'organic_results',
serpExtractor: (content) => {

View File

@@ -0,0 +1,35 @@
interface SerperResult {
title: string,
link: string,
position: number,
}
const serper:ScraperSettings = {
id: 'serper',
name: 'Serper.dev',
website: 'serper.dev',
allowsCity: true,
scrapeURL: (keyword, settings, countryData) => {
const country = keyword.country || 'US';
const lang = countryData[country][2];
return `https://google.serper.dev/search?q=${encodeURI(keyword.keyword)}&gl=${country}&hl=${lang}&num=100&apiKey=${settings.scaping_api}`;
},
resultObjectKey: 'organic',
serpExtractor: (content) => {
const extractedResult = [];
const results: SerperResult[] = (typeof content === 'string') ? JSON.parse(content) : content as SerperResult[];
for (const { link, title, position } of results) {
if (title && link) {
extractedResult.push({
title,
url: link,
position,
});
}
}
return extractedResult;
},
};
export default serper;

View File

@@ -1,3 +1,5 @@
import countries from '../../utils/countries';
interface SpaceSerpResult {
title: string,
link: string,
@@ -9,10 +11,14 @@ const spaceSerp:ScraperSettings = {
id: 'spaceSerp',
name: 'Space Serp',
website: 'spaceserp.com',
allowsCity: true,
scrapeURL: (keyword, settings, countryData) => {
const country = keyword.country || 'US';
const countryName = countries[country][0];
const location = keyword.city ? `&location=${encodeURI(`${keyword.city},${countryName}`)}` : '';
const device = keyword.device === 'mobile' ? '&device=mobile' : '';
const lang = countryData[country][2];
return `https://api.spaceserp.com/google/search?apiKey=${settings.scaping_api}&q=${encodeURI(keyword.keyword)}&pageSize=100&gl=${country}&hl=${lang}${keyword.device === 'mobile' ? '&device=mobile' : ''}&resultBlocks=`;
return `https://api.spaceserp.com/google/search?apiKey=${settings.scaping_api}&q=${encodeURI(keyword.keyword)}&pageSize=100&gl=${country}&hl=${lang}${location}${device}&resultBlocks=`;
},
resultObjectKey: 'organic_results',
serpExtractor: (content) => {

View File

@@ -0,0 +1,41 @@
import countries from '../../utils/countries';
interface ValueSerpResult {
title: string,
link: string,
position: number,
domain: string,
}
const valueSerp:ScraperSettings = {
id: 'valueserp',
name: 'Value Serp',
website: 'valueserp.com',
allowsCity: true,
scrapeURL: (keyword, settings, countryData) => {
const country = keyword.country || 'US';
const countryName = countries[country][0];
const location = keyword.city ? `&location=${encodeURI(`${keyword.city},${countryName}`)}` : '';
const device = keyword.device === 'mobile' ? '&device=mobile' : '';
const lang = countryData[country][2];
console.log(`https://api.valueserp.com/search?api_key=${settings.scaping_api}&q=${encodeURI(keyword.keyword)}&gl=${country}&hl=${lang}${device}${location}&num=100&output=json&include_answer_box=false&include_advertiser_info=false`);
return `https://api.valueserp.com/search?api_key=${settings.scaping_api}&q=${encodeURI(keyword.keyword)}&gl=${country}&hl=${lang}${device}${location}&num=100&output=json&include_answer_box=false&include_advertiser_info=false`;
},
resultObjectKey: 'organic_results',
serpExtractor: (content) => {
const extractedResult = [];
const results: ValueSerpResult[] = (typeof content === 'string') ? JSON.parse(content) : content as ValueSerpResult[];
for (const result of results) {
if (result.title && result.link) {
extractedResult.push({
title: result.title,
url: result.link,
position: result.position,
});
}
}
return extractedResult;
},
};
export default valueSerp;

View File

@@ -19,6 +19,19 @@ export async function fetchDomains(router: NextRouter, withStats:boolean): Promi
return res.json();
}
export async function fetchDomain(router: NextRouter, domainName: string): Promise<{domain: DomainType}> {
if (!domainName) { throw new Error('No Domain Name Provided!'); }
const res = await fetch(`${window.location.origin}/api/domain?domain=${domainName}`, { method: 'GET' });
if (res.status >= 400 && res.status < 600) {
if (res.status === 401) {
console.log('Unauthorized!!');
router.push('/login');
}
throw new Error('Bad response from server');
}
return res.json();
}
export async function fetchDomainScreenshot(domain: string, screenshot_key:string, forceFetch = false): Promise<string | false> {
const domainThumbsRaw = localStorage.getItem('domainThumbs');
const domThumbs = domainThumbsRaw ? JSON.parse(domainThumbsRaw) : {};
@@ -53,6 +66,14 @@ export function useFetchDomains(router: NextRouter, withStats:boolean = false) {
return useQuery('domains', () => fetchDomains(router, withStats));
}
export function useFetchDomain(router: NextRouter, domainName:string, onSuccess: Function) {
return useQuery('domain', () => fetchDomain(router, domainName), {
onSuccess: async (data) => {
console.log('Domain Loaded!!!', data.domain);
onSuccess(data.domain);
} });
}
export function useAddDomain(onSuccess:Function) {
const router = useRouter();
const queryClient = useQueryClient();
@@ -89,10 +110,11 @@ export function useUpdateDomain(onSuccess:Function) {
const headers = new Headers({ 'Content-Type': 'application/json', Accept: 'application/json' });
const fetchOpts = { method: 'PUT', headers, body: JSON.stringify(domainSettings) };
const res = await fetch(`${window.location.origin}/api/domains?domain=${domain.domain}`, fetchOpts);
const responseObj = await res.json();
if (res.status >= 400 && res.status < 600) {
throw new Error('Bad response from server');
throw new Error(responseObj?.error || 'Bad response from server');
}
return res.json();
return responseObj;
}, {
onSuccess: async () => {
console.log('Settings Updated!!!');
@@ -100,8 +122,8 @@ export function useUpdateDomain(onSuccess:Function) {
onSuccess();
queryClient.invalidateQueries(['domains']);
},
onError: () => {
console.log('Error Updating Domain Settings!!!');
onError: (error) => {
console.log('Error Updating Domain Settings!!!', error);
toast('Error Updating Domain Settings', { icon: '⚠️' });
},
});

View File

@@ -2,16 +2,21 @@ import toast from 'react-hot-toast';
import { NextRouter } from 'next/router';
import { useMutation, useQuery, useQueryClient } from 'react-query';
export const fetchKeywords = async (router: NextRouter) => {
if (!router.query.slug) { return []; }
const res = await fetch(`${window.location.origin}/api/keywords?domain=${router.query.slug}`, { method: 'GET' });
export const fetchKeywords = async (router: NextRouter, domain: string) => {
if (!domain) { return []; }
const res = await fetch(`${window.location.origin}/api/keywords?domain=${domain}`, { method: 'GET' });
return res.json();
};
export function useFetchKeywords(router: NextRouter, setKeywordSPollInterval?:Function, keywordSPollInterval:undefined|number = undefined) {
export function useFetchKeywords(
router: NextRouter,
domain: string,
setKeywordSPollInterval?:Function,
keywordSPollInterval:undefined|number = undefined,
) {
const { data: keywordsData, isLoading: keywordsLoading, isError } = useQuery(
['keywords', router.query.slug],
() => fetchKeywords(router),
['keywords', domain],
() => fetchKeywords(router, domain),
{
refetchInterval: keywordSPollInterval,
onSuccess: (data) => {

32
types.d.ts vendored
View File

@@ -15,6 +15,7 @@ type DomainType = {
scVisits?: number,
scImpressions?: number,
scPosition?: number,
search_console?: string,
}
type KeywordHistory = {
@@ -39,6 +40,7 @@ type KeywordType = {
lastUpdateError: {date: string, error: string, scraper: string} | false,
scData?: KeywordSCData,
uid?: string
city?: string
}
type KeywordLastResult = {
@@ -61,9 +63,17 @@ type countryCodeData = {
[ISO:string] : string
}
type DomainSearchConsole = {
property_type: 'domain' | 'url',
url: string,
client_email:string,
private_key:string,
}
type DomainSettings = {
notification_interval: string,
notification_emails: string,
search_console?: DomainSearchConsole
}
type SettingsType = {
@@ -77,14 +87,17 @@ type SettingsType = {
smtp_port: string,
smtp_username?: string,
smtp_password?: string,
search_console_integrated?: boolean,
available_scapers?: Array,
available_scapers?: { label: string, value: string, allowsCity?: boolean }[],
scrape_interval?: string,
scrape_delay?: string,
scrape_retry?: boolean,
failed_queue?: string[]
version?: string,
screenshot_key?: string,
search_console: boolean,
search_console_client_email: string,
search_console_private_key: string,
search_console_integrated?: boolean,
}
type KeywordSCDataChild = {
@@ -108,7 +121,8 @@ type KeywordAddPayload = {
device: string,
country: string,
domain: string,
tags: string,
tags?: string,
city?:string
}
type SearchAnalyticsRawItem = {
@@ -177,11 +191,23 @@ type scraperExtractedItem = {
position: number,
}
interface ScraperSettings {
/** A Unique ID for the Scraper. eg: myScraper */
id:string,
/** The Name of the Scraper */
name:string,
/** The Website address of the Scraper */
website:string,
/** The result object's key that contains the results of the scraped data. For example,
* if your scraper API the data like this `{scraped:[item1,item2..]}` the resultObjectKey should be "scraped" */
resultObjectKey: string,
/** If the Scraper allows setting a perices location or allows city level scraping set this to true. */
allowsCity?: boolean,
/** Set your own custom HTTP header properties when making the scraper API request.
* The function should return an object that contains all the header properties you want to pass to API request's header.
* Example: `{'Cache-Control': 'max-age=0', 'Content-Type': 'application/json'}` */
headers?(keyword:KeywordType, settings: SettingsType): Object,
/** Construct the API URL for scraping the data through your Scraper's API */
scrapeURL?(keyword:KeywordType, settings:SettingsType, countries:countryData): string,
/** Custom function to extract the serp result from the scraped data. The extracted data should be @return {scraperExtractedItem[]} */
serpExtractor?(content:string): scraperExtractedItem[],
}

View File

@@ -33,3 +33,14 @@ export const isValidDomain = (domain:string): boolean => {
return isValid;
};
export const isValidUrl = (str: string) => {
let url;
try {
url = new URL(str);
} catch (e) {
return false;
}
return url.protocol === 'http:' || url.protocol === 'https:';
};

View File

@@ -187,15 +187,17 @@ export const extractScrapedResult = (content: string, device: string): SearchRes
/**
* Find in the domain's position from the extracted search result.
* @param {string} domain - Domain Name to look for.
* @param {string} domainURL - URL Name to look for.
* @param {SearchResult[]} result - The search result array extracted from the Google Search result.
* @returns {SERPObject}
*/
export const getSerp = (domain:string, result:SearchResult[]) : SERPObject => {
if (result.length === 0 || !domain) { return { postion: 0, url: '' }; }
export const getSerp = (domainURL:string, result:SearchResult[]) : SERPObject => {
if (result.length === 0 || !domainURL) { return { postion: 0, url: '' }; }
const URLToFind = new URL(domainURL.includes('https://') ? domainURL : `https://${domainURL}`);
const theURL = URLToFind.hostname + URLToFind.pathname;
const foundItem = result.find((item) => {
const itemDomain = item.url.replace('www.', '').match(/^(?:https?:)?(?:\/\/)?([^/?]+)/i);
return itemDomain && itemDomain.includes(domain.replace('www.', ''));
const itemURL = new URL(item.url.includes('https://') ? item.url : `https://${item.url}`);
return theURL === itemURL.hostname + itemURL.pathname || `${theURL}/` === itemURL.hostname + itemURL.pathname;
});
return { postion: foundItem ? foundItem.position : 0, url: foundItem && foundItem.url ? foundItem.url : '' };
};

View File

@@ -1,4 +1,5 @@
import { auth, searchconsole_v1 } from '@googleapis/searchconsole';
import Cryptr from 'cryptr';
import { readFile, writeFile, unlink } from 'fs/promises';
import { getCountryCodeFromAlphaThree } from './countries';
@@ -7,22 +8,32 @@ export type SCDomainFetchError = {
errorMsg: string,
}
type SCAPISettings = { client_email: string, private_key: string }
type fetchConsoleDataResponse = SearchAnalyticsItem[] | SearchAnalyticsStat[] | SCDomainFetchError;
/**
* function that retrieves data from the Google Search Console API based on the provided domain name, number of days, and optional type.
* @param {string} domainName - The domain name for which you want to fetch search console data.
* @param {number} days - The `days` parameter is the number of days of data you want to fetch from the Search Console.
* @param {string} [type] - The `type` parameter is an optional parameter that specifies the type of data to fetch from the Search Console API.
* Retrieves data from the Google Search Console API based on the provided domain name, number of days, and optional type.
* @param {DomainType} domain - The domain for which you want to fetch search console data.
* @param {number} days - number of days of data you want to fetch from the Search Console.
* @param {string} [type] - (optional) specifies the type of data to fetch from the Search Console.
* @param {SCAPISettings} [api] - (optional) specifies the Seach Console API Information.
* @returns {Promise<fetchConsoleDataResponse>}
*/
const fetchSearchConsoleData = async (domainName:string, days:number, type?:string): Promise<fetchConsoleDataResponse> => {
if (!domainName) return { error: true, errorMsg: 'Domain Not Provided!' };
const fetchSearchConsoleData = async (domain:DomainType, days:number, type?:string, api?:SCAPISettings): Promise<fetchConsoleDataResponse> => {
if (!domain) return { error: true, errorMsg: 'Domain Not Provided!' };
if (!api?.private_key || !api?.client_email) return { error: true, errorMsg: 'Search Console API Data Not Avaialable.' };
const domainName = domain.domain;
const defaultSCSettings = { property_type: 'domain', url: '', client_email: '', private_key: '' };
const domainSettings = domain.search_console ? JSON.parse(domain.search_console) : defaultSCSettings;
const sCPrivateKey = api?.private_key || process.env.SEARCH_CONSOLE_PRIVATE_KEY || '';
const sCClientEmail = api?.client_email || process.env.SEARCH_CONSOLE_CLIENT_EMAIL || '';
try {
const authClient = new auth.GoogleAuth({
credentials: {
private_key: process.env.SEARCH_CONSOLE_PRIVATE_KEY ? process.env.SEARCH_CONSOLE_PRIVATE_KEY.replaceAll('\\n', '\n') : '',
client_email: process.env.SEARCH_CONSOLE_CLIENT_EMAIL ? process.env.SEARCH_CONSOLE_CLIENT_EMAIL : '',
private_key: (sCPrivateKey).replaceAll('\\n', '\n'),
client_email: (sCClientEmail || '').trim(),
},
scopes: [
'https://www.googleapis.com/auth/webmasters.readonly',
@@ -51,7 +62,8 @@ const fetchSearchConsoleData = async (domainName:string, days:number, type?:stri
};
}
const res = client.searchanalytics.query({ siteUrl: `sc-domain:${domainName}`, requestBody });
const siteUrl = domainSettings.property_type === 'url' && domainSettings.url ? domainSettings.url : `sc-domain:${domainName}`;
const res = client.searchanalytics.query({ siteUrl, requestBody });
const resData:any = (await res).data;
let finalRows = resData.rows ? resData.rows.map((item:SearchAnalyticsRawItem) => parseSearchConsoleItem(item, domainName)) : [];
@@ -70,26 +82,29 @@ const fetchSearchConsoleData = async (domainName:string, days:number, type?:stri
}
return finalRows;
} catch (error:any) {
} catch (err:any) {
const qType = type === 'stats' ? '(stats)' : `(${days}days)`;
console.log(`[ERROR] Search Console API Error for ${domainName} ${qType} : `, error?.response?.status, error?.response?.statusText);
return { error: true, errorMsg: `${error?.response?.status}: ${error?.response?.statusText}` };
const errorMsg = err?.response?.status && `${err?.response?.statusText}. ${err?.response?.data?.error_description}`;
console.log(`[ERROR] Search Console API Error for ${domainName} ${qType} : `, errorMsg || err?.code);
// console.log('SC ERROR :', err);
return { error: true, errorMsg: errorMsg || err?.code };
}
};
/**
* The function fetches search console data for a given domain and returns it in a structured format.
* @param {string} domain - The `domain` parameter is a string that represents the domain for which we
* @param {DomainType} domain - The `domain` parameter is a Domain object that represents the domain for which we
* want to fetch search console data.
* @returns The function `fetchDomainSCData` is returning a Promise that resolves to an object of type
* `SCDomainDataType`.
*/
export const fetchDomainSCData = async (domain:string): Promise<SCDomainDataType> => {
export const fetchDomainSCData = async (domain:DomainType, scAPI?: SCAPISettings): Promise<SCDomainDataType> => {
const days = [3, 7, 30];
const scDomainData:SCDomainDataType = { threeDays: [], sevenDays: [], thirtyDays: [], lastFetched: '', lastFetchError: '', stats: [] };
if (domain) {
if (domain.domain && scAPI) {
const theDomain = domain;
for (const day of days) {
const items = await fetchSearchConsoleData(domain, day);
const items = await fetchSearchConsoleData(theDomain, day, undefined, scAPI);
scDomainData.lastFetched = new Date().toJSON();
if (Array.isArray(items)) {
if (day === 3) scDomainData.threeDays = items as SearchAnalyticsItem[];
@@ -99,11 +114,11 @@ export const fetchDomainSCData = async (domain:string): Promise<SCDomainDataType
scDomainData.lastFetchError = items.errorMsg;
}
}
const stats = await fetchSearchConsoleData(domain, 30, 'stat');
const stats = await fetchSearchConsoleData(theDomain, 30, 'stat', scAPI);
if (stats && Array.isArray(stats) && stats.length > 0) {
scDomainData.stats = stats as SearchAnalyticsStat[];
}
await updateLocalSCData(domain, scDomainData);
await updateLocalSCData(domain.domain, scDomainData);
}
return scDomainData;
@@ -139,9 +154,9 @@ export const integrateKeywordSCData = (keyword: KeywordType, SCData:SCDomainData
const ctr:any = { yesterday: 0, threeDays: 0, sevenDays: 0, thirtyDays: 0, avgSevenDays: 0, avgThreeDays: 0, avgThirtyDays: 0 };
const position:any = { yesterday: 0, threeDays: 0, sevenDays: 0, thirtyDays: 0, avgSevenDays: 0, avgThreeDays: 0, avgThirtyDays: 0 };
const threeDaysData = SCData.threeDays.find((item:SearchAnalyticsItem) => item.uid === kuid) || {};
const SevenDaysData = SCData.sevenDays.find((item:SearchAnalyticsItem) => item.uid === kuid) || {};
const ThirdyDaysData = SCData.thirtyDays.find((item:SearchAnalyticsItem) => item.uid === kuid) || {};
const threeDaysData = SCData?.threeDays?.find((item:SearchAnalyticsItem) => item.uid === kuid) || {};
const SevenDaysData = SCData?.sevenDays?.find((item:SearchAnalyticsItem) => item.uid === kuid) || {};
const ThirdyDaysData = SCData?.thirtyDays?.find((item:SearchAnalyticsItem) => item.uid === kuid) || {};
const totalData:any = { threeDays: threeDaysData, sevenDays: SevenDaysData, thirtyDays: ThirdyDaysData };
Object.keys(totalData).forEach((dataKey) => {
@@ -164,16 +179,71 @@ export const integrateKeywordSCData = (keyword: KeywordType, SCData:SCDomainData
return { ...keyword, scData: finalSCData };
};
/**
* Retrieves the Search Console API information for a given domain.
* @param {DomainType} domain - The `domain` parameter is of type `DomainType`, which represents a
* domain object. It likely contains information about a specific domain, such as its name, search
* console settings, etc.
* @returns an object of type `SCAPISettings`.
*/
export const getSearchConsoleApiInfo = async (domain: DomainType): Promise<SCAPISettings> => {
const scAPIData = { client_email: '', private_key: '' };
// Check if the Domain Has the API Data
const domainSCSettings = domain.search_console && JSON.parse(domain.search_console);
if (domainSCSettings && domainSCSettings.private_key) {
if (!domainSCSettings.private_key.includes('BEGIN PRIVATE KEY')) {
const cryptr = new Cryptr(process.env.SECRET as string);
scAPIData.client_email = domainSCSettings.client_email ? cryptr.decrypt(domainSCSettings.client_email) : '';
scAPIData.private_key = domainSCSettings.private_key ? cryptr.decrypt(domainSCSettings.private_key) : '';
} else {
scAPIData.client_email = domainSCSettings.client_email;
scAPIData.private_key = domainSCSettings.private_key;
}
}
// Check if the App Settings Has the API Data
if (!scAPIData?.private_key) {
const settingsRaw = await readFile(`${process.cwd()}/data/settings.json`, { encoding: 'utf-8' });
const settings: SettingsType = settingsRaw ? JSON.parse(settingsRaw) : {};
const cryptr = new Cryptr(process.env.SECRET as string);
scAPIData.client_email = settings.search_console_client_email ? cryptr.decrypt(settings.search_console_client_email) : '';
scAPIData.private_key = settings.search_console_private_key ? cryptr.decrypt(settings.search_console_private_key) : '';
}
if (!scAPIData?.private_key && process.env.SEARCH_CONSOLE_PRIVATE_KEY && process.env.SEARCH_CONSOLE_CLIENT_EMAIL) {
scAPIData.client_email = process.env.SEARCH_CONSOLE_CLIENT_EMAIL;
scAPIData.private_key = process.env.SEARCH_CONSOLE_PRIVATE_KEY;
}
return scAPIData;
};
/**
* Checks if the provided domain level Google Search Console API info is valid.
* @param {DomainType} domain - The domain that represents the domain for which the SC API info is being checked.
* @returns an object of type `{ isValid: boolean, error: string }`.
*/
export const checkSerchConsoleIntegration = async (domain: DomainType): Promise<{ isValid: boolean, error: string }> => {
const res = { isValid: false, error: '' };
const { client_email = '', private_key = '' } = domain?.search_console ? JSON.parse(domain.search_console) : {};
const response = await fetchSearchConsoleData(domain, 3, undefined, { client_email, private_key });
if (Array.isArray(response)) { res.isValid = true; }
if ((response as SCDomainFetchError)?.errorMsg) { res.error = (response as SCDomainFetchError).errorMsg; }
return res;
};
/**
* The function reads and returns the domain-specific data stored in a local JSON file.
* @param {string} domain - The `domain` parameter is a string that represents the domain for which the SC data is being read.
* @returns {Promise<SCDomainDataType>}
*/
export const readLocalSCData = async (domain:string): Promise<SCDomainDataType> => {
const filePath = `${process.cwd()}/data/SC_${domain}.json`;
const currentQueueRaw = await readFile(filePath, { encoding: 'utf-8' }).catch(async () => { await updateLocalSCData(domain); return '{}'; });
const domainSCData = JSON.parse(currentQueueRaw);
return domainSCData;
export const readLocalSCData = async (domain:string): Promise<SCDomainDataType|false> => {
try {
const filePath = `${process.cwd()}/data/SC_${domain.replaceAll('/', '-')}.json`;
const currentQueueRaw = await readFile(filePath, { encoding: 'utf-8' }).catch(async () => { await updateLocalSCData(domain); return '{}'; });
const domainSCData = JSON.parse(currentQueueRaw);
return domainSCData;
} catch (error) {
return false;
}
};
/**
@@ -183,10 +253,14 @@ export const readLocalSCData = async (domain:string): Promise<SCDomainDataType>
* @returns {Promise<SCDomainDataType|false>}
*/
export const updateLocalSCData = async (domain:string, scDomainData?:SCDomainDataType): Promise<SCDomainDataType|false> => {
const filePath = `${process.cwd()}/data/SC_${domain}.json`;
const emptyData:SCDomainDataType = { threeDays: [], sevenDays: [], thirtyDays: [], lastFetched: '', lastFetchError: '' };
await writeFile(filePath, JSON.stringify(scDomainData || emptyData), { encoding: 'utf-8' }).catch((err) => { console.log(err); });
return scDomainData || emptyData;
try {
const filePath = `${process.cwd()}/data/SC_${domain.replaceAll('/', '-')}.json`;
const emptyData:SCDomainDataType = { threeDays: [], sevenDays: [], thirtyDays: [], lastFetched: '', lastFetchError: '' };
await writeFile(filePath, JSON.stringify(scDomainData || emptyData), { encoding: 'utf-8' }).catch((err) => { console.log(err); });
return scDomainData || emptyData;
} catch (error) {
return false;
}
};
/**
@@ -195,7 +269,7 @@ export const updateLocalSCData = async (domain:string, scDomainData?:SCDomainDat
* @returns {Promise<boolean>} - Returns true if file was removed, else returns false.
*/
export const removeLocalSCData = async (domain:string): Promise<boolean> => {
const filePath = `${process.cwd()}/data/SC_${domain}.json`;
const filePath = `${process.cwd()}/data/SC_${domain.replaceAll('/', '-')}.json`;
try {
await unlink(filePath);
return true;